For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

---

The first is the UDP we want, HTTP/3 for our requests. The second is the UDP we never asked for, the WebRTC traffic that a browser generates on its own, which can leak around the proxy and expose our real IP. But supporting UDP and actually getting a client to route its UDP through the proxy are two different problems, and the result changes a lot depending on the client. An HTTP client, a browser doing HTTP/3, and a browser doing WebRTC all behave differently, so we tested all three, end to end, to see where a UDP proxy does the work for us and where we still have to step in on the client side.

A quick recap of HTTP/2 and HTTP/3

We already covered the request side of HTTP/3 in Faster Web Scraping with HTTP/3, so let’s just recap what matters for proxies. HTTP/1.1 and HTTP/2 run on TCP. HTTP/3 drops TCP completely and runs on QUIC, which is built on UDP and merges the transport and TLS handshakes into one. The part that matters for us is the transport: a proxy built to relay TCP has nothing to grab onto in a QUIC connection, since there’s no TCP stream to relay.

Adoption is real, but it’s easy to overstate. If we look at traffic, HTTP/3 is around a fifth of all requests, while HTTP/2 is still the majority at almost half, and this split has stayed roughly the same through 2026 (Cloudflare Radar). If we look at site support instead, the number goes up: around 38% of sites advertise HTTP/3 via Alt-Svc (W3Techs). Italy, where our test machine sits, is one of the higher HTTP/3 markets at about 30%. So HTTP/3 is a serious, well-supported protocol, not some niche thing, but it hasn’t replaced HTTP/2 yet, and most real traffic is still on TCP.

HTTP/3 and QUIC run on UDP, so the proxy layer has to carry it too. anyIP gives compatible clients production-ready SOCKS5 UDP ASSOCIATE across its residential and mobile network.

Explore anyIP UDP Proxies

The sales pitch follows naturally from here. If real browsers increasingly speak HTTP/3 and our scraper can’t, that difference becomes a signal, so we’d want to speak HTTP/3 too, and to do that through a proxy, the proxy has to carry UDP.

This is exactly what some providers are now selling. They offer UDP support, usually as a beta over SOCKS5, and they pitch it as a stealth feature: the idea is that most residential traffic is QUIC these days, so a client stuck on TCP stands out. There are two parts of this claim worth checking before relying on it, the stealth side and the tooling side, so we tested both, using a provider that carries UDP and one that doesn’t.

UDP support and the stealth question

Switching to HTTP/3 doesn’t remove protocol fingerprinting, it just moves it somewhere else. Detection happens at three layers, and moving from TCP to QUIC affects each one in a different way.

The TCP/IP stack fingerprint, meaning the SYN options, window sizes and TTL that give away the OS, really does disappear, because there’s no TCP handshake anymore. That’s the only layer that goes away, though. The TLS fingerprint stays exactly as it was, because QUIC still does a TLS 1.3 handshake, just wrapped inside QUIC instead of running over TCP. The ClientHello with its cipher suites and extensions is still there, and JA4 was designed to fingerprint the QUIC variant on purpose (Scrapfly QUIC/HTTP3 fingerprint). The HTTP/2 fingerprint, the SETTINGS frames, window updates and pseudo-header order, doesn’t get erased either. It just gets replaced by an HTTP/3 fingerprint built from QUIC transport parameters.

And those QUIC transport parameters are basically the same thing we were trying to get away from. Scrapfly says it clearly: they “work like HTTP/2 SETTINGS frames but at the transport layer.” Values like max_udp_payload_size and initial_max_data, and the order they come in, are different between Chrome’s quiche, Firefox, Go’s quic-go and ngtcp2, and that difference is itself a fingerprint (Scrapfly HTTP/2 and HTTP/3 fingerprinting). QUIC even adds new signals that have no TCP equivalent at all, like 0-RTT behavior and connection migration support.

So the layer that actually gets scrapers blocked today is TLS and HTTP/2 fingerprinting, which Cloudflare, Akamai and DataDome score together as JA4 and JA4H. That’s a TCP-side problem, and it’s one we already solve with a client that imitates a browser’s handshake. HTTP/3 fingerprinting is growing but still immature. Scrapfly is pretty direct about it: it “adds signals on top of” HTTP/2 instead of replacing it, real Chrome on HTTP/2 is still accepted on the vast majority of sites, and the scraper-on-HTTP/2 versus user-on-HTTP/3 gap is only “starting to become a signal on Cloudflare’s premium tier.” There’s also a catch worth keeping in mind: a scraper that speaks HTTP/3 through a non-browser QUIC stack shows a QUIC fingerprint that matches no real browser, so it can stand out just as easily as it blends in. In other words, a UDP proxy is what gets us onto HTTP/3 in the first place, but the stealth benefit only pays off if the client presents a QUIC fingerprint that a real browser would. The proxy just carries the traffic; making it look right is still our job.

Check the TWSC YouTube Channel

The tooling has not caught up yet

The other half of the pitch is the tooling. The most common client for browserless scraping is curl_cffi, so the real question is whether it can actually do HTTP/3 in the setup a scraper normally runs in. We tested it (run_curl_cffi_http3.py), and you can find the code, as always, on our GitHub repository available for our paid readers, under the folder 109.UDP

The library is compiled with the QUIC stack, ngtcp2 and nghttp3, and on a plain direct request it does reach HTTP/3. It even reaches HTTP/3 with a full Chrome fingerprint if we use the low-level interface, so impersonation and HTTP/3 aren’t really in conflict, even if the first error makes it look that way.

high-level requests API:
  plain, no impersonation, V3ONLY        -> HTTP/3
  Chrome impersonation, V3ONLY           -> ERROR: QUIC needs at least TLS version 1.3
  Chrome impersonation, HTTP proxy, V3   -> ERROR: HTTP/3 is not supported over an HTTP proxy
  Chrome impersonation, SOCKS5, V3       -> ERROR: HTTP/3 is not supported over a SOCKS proxy
low-level Curl API (same impersonation):
  impersonate chrome + V3ONLY            -> HTTP/3

Need help with your scraping project?

That middle error in the high-level API is just a TLS-version conflict inside the requests wrapper, not a real limitation, since the low-level call combines the Chrome fingerprint with HTTP/3 and gets there fine. The lines that really matter are the proxy ones. curl refuses HTTP/3 over a proxy completely, both HTTP and SOCKS, and it refuses before it even connects, so this is curl saying no, not the proxy. It happens even with a proxy that could carry the UDP. And since a scraper always runs through a proxy, in practice curl_cffi ends up being HTTP/2 over TCP, and a UDP-capable proxy doesn’t change that at all.

So this is the honest picture on the request side. A proxy that carries UDP does its job for a client that tunnels its own UDP, but the popular HTTP clients refuse HTTP/3 over a proxy right now, so the benefit is stuck waiting for the tooling to catch up. The proxy capability is arriving before the clients that could use it, which is good to know before expecting HTTP/3 to flow through a scraper just because the proxy supports UDP.

The WebRTC leak, where UDP really bites

Read more

However, web scraping can also serve as a foundation for building the data infrastructure needed to better understand national markets and societal conditions.

In this blog post, I’ll show real-world examples of how web scraping is already used in the public sector, and how institutions could leverage it to improve policies and deliver higher-quality services to citizens.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

---

Why Web Scraping Matters for Public Institutions

Public institutions have access to statistical datasets from official sources. These are surely relevant, but they tend to provide a picture of the market only after events have already happened, sometimes months or even a year later.

That delay creates a challenge. After all, institutions need timely information to understand citizens’ current needs and respond effectively. Another issue is that relevant data is often scattered across many sources, such as different websites, web portals, and marketplaces.

This is where web scraping can help! By collecting large volumes of publicly available data from multiple sites, institutions can complement and enrich official statistics.

Data collection is only the first step, though. The real value comes from building a complete data pipeline that includes cleaning, deduplication, aggregation, geolocation, statistical modeling, and interpretation.

At Prague Crawl 2026, Bayernheimerová Klára presented a compelling example of this approach. Her talk showed how the Czech Ministry of Finance relies on rental listing data collected by an external provider from several real estate portals and processes it through an end-to-end pipeline based on a dedicated statistical methodology.

As I’ll present shortly, the result is a reliable source of insights that supports housing policy decisions and affordable housing programs across the country, while also providing a valuable service to citizens.

What’s important to understand is that the goal isn’t simply to collect raw data, but to power services such as interactive maps, calculators, and dashboards that are useful to both public institutions and citizens.

Case Study: Czech Rental Market Intelligence System

To see an actual application of web scraping in the public sector, I’ll now present the initiative developed by the Czech Ministry of Finance for rental price analysis across Czechia.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The Motivation Behind the Project

Just like in many other European countries, housing affordability has become a major social and economic challenge in the Czech Republic. Housing supply has struggled to keep up with demand, causing both property prices and rents to rise steadily across the country.

To confront this issue, the Czech Ministry of Finance needed detailed and up-to-date information on local rental markets. The initiative was launched specifically to fill that gap, creating a web data analysis process that gives a continuously updated view of rental prices across the entire country.

The objective is to provide a reliable foundation for housing policy, affordable housing programs, and public-facing tools that help citizens better resonate with local rental markets.

Web Data Sourcing

The data for this initiative has been collected using Apify, one of the largest marketplaces of ready-made tools for web scraping, automation, and AI.

If you aren’t familiar with Apify, it provides over 41,000 ready-made serverless cloud programs (called Actors) to automate a wide range of tasks, including web scraping across thousands of different domains.

These Actors are developed and maintained by the community (and in some cases by Apify itself) and run on Apify’s cloud infrastructure. You can use them directly through Apify Console via a no-code interface, or call them programmatically via API. They can also be integrated into workflows such as n8n, Make, Zapier, or AI agents via MCP.

Two main reasons influenced the choice of Apify:

1. It’s a Czech-based company.

2. It enables the Ministry of Finance to gather large-scale, up-to-date information on rental listings without maintaining scraping infrastructure in-house.

In general, a solution like Apify lowers the barrier for public-sector teams, as these may have limited engineering capacity.

Downstream Data Pipeline

At a high level, this is the downstream data pipeline implemented by the Czech Ministry of Finance:

1. Data ingestion: Collects rental listings via Apify-based web scraping from multiple Czech real estate portals.

2. Data cleaning: Sorts data and removes duplicates, incorrect entries, and inconsistencies to ensure it is up to date, accurate, and free from deviations or distortions, following the procedure defined in Decree No. 456/2024 Coll.

3. Geolocation and aggregation: Standardizes addresses, assigns cadastral units, and enriches listings with attributes such as size, amenities, and building type.

4. Statistical modelling: Uses hedonic regression with spatial and temporal weighting to estimate underlying rental price levels.

5. Output generation: Presents results as interactive price maps and calculators.

Note: The pipeline also includes data update and recycling via a rolling window approach. This process adds new listings while retiring outdated observations to keep the dataset current.

To make rental prices comparable across different locations, the methodology is based on a reference apartment. That represents a standardized apartment profile with predefined characteristics, such as size, type, furnishing level, etc.

This approach reduces differences caused by individual property features and enables more consistent comparisons of rental price levels across municipalities and cadastral areas.

Produced Output and Tools

The Czech Rental Housing Price Map consists of two main solutions:

• Interactive rent price map: Provides estimated rental prices at the level of municipalities and cadastral areas across the Czech Republic.

• Market rent calculator: A practical tool that calculates a statistical estimate of rental price levels based on a standardized apartment profile and selected property characteristics.

Important: The two solutions aren’t intended to determine the market rent of a specific apartment. Instead, they return statistical estimates based on a standardized reference apartment. So, they’re intended primarily for market monitoring, regional comparisons, and housing policy analysis.

Interactive rent price map

The interactive rent price map shows minimum, maximum, and median rents for each area based on market listings, adjusted per square meter for a standard unfurnished reference apartment. It covers four size categories from 1+kk/1+1 to 4+kk/4+1.

Note that the map is interactive, and you can zoom in and out to explore individual cadastral areas. It’s also updated four times per year to ensure current data.

Market rent calculator

The market rent calculator is available as a form on the Czech government website. It works as follows:

1. Select the territorial unit where you want to estimate the rent.

2. Choose the size category of the apartment.

3. Enter the floor area of the apartment (if you don’t know it, the calculator will automatically use a default value for the selected location).

4. Indicate whether the apartment is in a new building.

5. Optionally specify whether the building uses non-standard construction materials (e.g., other than brick or panel).

6. Optionally select additional features such as a terrace, furniture, or an assigned parking/garage space.

7. Click “Calculate rent” to obtain an estimate of the monthly market rent (CZK) and the corresponding price level, which represents the final estimated rent for the selected apartment profile.

Impact

The project provides a systematic and regularly updated overview of market rental prices across the entire Czech Republic, supporting consistent monitoring of housing market developments over time and across regions.

Beyond market observation, the outputs serve as a key evidence base for housing policy design and housing programmes within public administration. For example, the State Investment Support Fund (SFPI) relies on these results when working with its affordable rental housing schemes.

Web Scraping in the Public Sector: Extending the Model

The project carried out by the Czech Ministry of Finance is just one example of a much broader pattern: how web scraping can be used as a foundation for modern public-sector data systems.

In particular, the same pipeline logic can be applied across many other domains. For example, in labor markets, web scraping can be used to collect job postings, salary ranges, and skill requirements from recruitment platforms. This can help governments identify regional skill shortages and design more targeted education or reskilling programmes.

Similarly, in consumer price monitoring, scraping can track grocery prices, housing costs, and essential goods across regions, enabling better inflation tracking and cost-of-living analysis. Other potential public domains that can benefit from this include energy consumption and transport accessibility.

When combined, those datasets become even more powerful. For instance, housing data, job market data, and grocery price data could be combined into a broader “liveability” index. That would help citizens and policymakers assess the overall affordability of living in a given region.

Check the TWSC YouTube Channel

Building a Job Market Choropleth for Czech Districts

In this section, I’ll guide you through building a job market choropleth across Czech districts. The idea is to show how web scraping, combined with a complete data pipeline, can be utilized for other scenarios beyond the previous example.

Important: The project below isn’t related to the initiative from the Czech Ministry of Finance and has been created purely for illustrative purposes.

Prerequisites

To follow this tutorial section, make sure you have:

• An Apify account (a free plan is sufficient).

• Python 3.11+ installed locally.

I’ll also assume you have a Python project set up locally with the following libraries installed:

pip install pandas geopandas geopy matplotlib

These are the required dependencies for this project and will be used as follows:

• pandas: Load, clean, and manipulate the scraped job posting data before geospatial analysis.

• geopandas: Handle geographic data, perform spatial joins, and create district-level maps.

• geopy: Convert job location addresses into geographic coordinates through geocoding.

• matplotlib: Visualize the results by generating a choropleth map showing job density by district.

Step #1: Access the Indeed Jobs Scraper Apify Actor

Just like in the Czech Ministry of Finance example, you can use Apify to collect the source data. This saves you from building and maintaining a complete job scraping pipeline from scratch.

For this project, we’ll use Indeed as the data source. Indeed has a Czech version, and most listings include the full office address where employees are expected to work.

In detail, I recommend using the Indeed Jobs Scraper [PPR] Actor. This automates the extraction of job titles, salaries, locations, company information, and job descriptions from Indeed.

To get started, log in to your Apify account and select the “Apify Store” option from the left-hand navigation menu. Then, search for “Indeed Jobs Scraper [PPR]”:

Click on the Actor card to reach its page. Great!

Step #2: Run the Job Scraping Task

On the Actor page, you’ll find an input form that lets you configure the scraper before running it in the cloud. Select “Czech Republic” as the “Country” and enter “Software engineer” in the “Query” field:

Note: In a real-world scenario, configure the Actor’s inputs according to your needs.

Next, toggle the option to avoid duplicates so that the Actor takes care of data deduplication for you:

Click the “Save & Start” button to launch the scraping task. The Actor will start running directly in Apify Console. As the scraper progresses, you’ll see the extracted job postings appear in real time. Be patient, as the process may take a few minutes depending on the number of matching listings.

Step #3: Explore the Output and Export the Results

Once the run completes, you’ll be able to explore the scraped dataset directly in Apify Console. As you can see, the Actor returns structured Indeed job data:

In this case, the Actor retrieved 63 job postings. That may seem like a small number, but keep in mind that the scraper targets only jobs published during the last 14 days. This helps ensure that the dataset reflects the current state of the job market.

Next, switch to the JSON view and select “All fields”. You’ll notice that each job posting includes a location object containing address information for the position:

That location data is exactly what you need to visualize job openings on a map and analyze their geographic distribution across Czech districts.

Finally, open the “Storage” tab, select the “JSON” export format, and click “Download” to export the scraped dataset:

A file with a name similar to dataset_indeed-scraper_2026-06-18_11-54-00-555.json will be downloaded. Rename it to jobs.json and place it in your Python project’s root directory.

Step #4: Download the Required Czech GeoJSON Data

To visualize the distribution of job openings across Czech districts, you’ll need a GeoJSON dataset containing the geographic boundaries of those districts.

Generally, that type of data is open and publicly available. For example, one possible source is the siwekm/czech-geojson repository, which provides GeoJSON files for various Czech administrative divisions:

Download the okresy.json file, which contains the geographic boundaries of Czech districts (okresy in Czech).

Note: Although the okresy administrative system was officially abolished in 2003, these districts are still used for statistical analysis. This makes them a good choice for visualizing the distribution of job opportunities across the country.

Once downloaded, add the okresy.json file to your Python project. At this point, your project structure should look similar to this:

├── jobs.json
├── okresy.json
└── main.py

Step #5: Visualize the Job Data on a Map

In your Python file, define the complete pipeline with the following code:

# pip install pandas geopandas geopy matplotlib

import json
from geopy.geocoders import Nominatim
from geopy.extra.rate_limiter import RateLimiter
import pandas as pd
import geopandas as gpd
import matplotlib.pyplot as plt

# Load the scraped Indeed job openings
with open("jobs.json", "r", encoding="utf-8") as f:
    jobs = json.load(f)

# Load the list of addresses from the job postings
addresses = []
for job in jobs:
    loc = job.get("location", {})

    # Prefer fullAddress, fallback to formattedAddressLong or formattedAddressShort
    address = (
        loc.get("fullAddress")
        or loc.get("formattedAddressLong")
        or loc.get("formattedAddressShort")
    )

    # Filter out full-remote jobs with no address
    if address and address.strip().lower() not in ["home office", "remote"]:
        addresses.append(address)

addresses = list(addresses)

# Initialize geocoder for Czech job locations
geolocator = Nominatim(user_agent="cz_job_map")
geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1)

# Geocode all job addresses into latitude/longitude points
rows = []
for address in addresses:
    loc = geocode(f"{address}, Czech Republic")

    # Keep only successfully geocoded results
    if loc:
        rows.append({
            "address": address,
            "lat": loc.latitude,
            "lon": loc.longitude
        })

# Convert geocoded results into a DataFrame
df = pd.DataFrame(rows)

# Convert DataFrame into GeoDataFrame with point geometries
points = gpd.GeoDataFrame(
    df,
    geometry=gpd.points_from_xy(df.lon, df.lat),
    crs="EPSG:4326"
)

# Load Czech administrative districts (okresy) from a local GeoJSON file
okresy = gpd.read_file("okresy.json")

# Align coordinate reference systems between datasets
if okresy.crs != points.crs:
    okresy = okresy.to_crs(points.crs)

# Assign each job point to its corresponding district
joined = gpd.sjoin(
    points,
    okresy,
    predicate="within",
    how="left"
)

# Count number of jobs per district
counts = (
    joined["id"]
    .value_counts()
    .reset_index()
)

counts.columns = ["id", "count"]


# Merge job counts back into district geometries
okresy = okresy.merge(
    counts,
    on="id",
    how="left"
)

# Replace missing values with 0 (districts with no jobs)
okresy["count"] = okresy["count"].fillna(0)

# Plot choropleth map with job counts per district
fig, ax = plt.subplots(figsize=(10, 8))

okresy.plot(
    column="count",
    cmap="Reds",
    edgecolor="black",
    linewidth=0.3,
    legend=True,
    vmin=0,
    ax=ax
)

# Add labels for districts with job activity
top = okresy[okresy["count"] > 0].sort_values("count", ascending=False)

for idx, row in top.iterrows():
    centroid = row.geometry.centroid

    ax.text(
        centroid.x,
        centroid.y,
        str(int(row["count"])),
        fontsize=9,
        ha="center",
        va="center",
        color=(lambda v: "white" if v > okresy["count"].max() * 0.5 else "black")(row["count"]),
        fontweight="bold"
    )

ax.set_title(
    "Software engineering job density on Indeed in the Czech Republic (last 14 days, by district)"
)
ax.axis("off")

plt.show() 

Need help with your scraping project?

That’s what the above script does:

1. Loads scraped job data from jobs.json and extracts valid physical addresses, filtering out remote or missing location entries.

2. Initializes the Nominatim geocoder with a rate limiter to safely convert addresses into latitude and longitude without exceeding request limits.

3. Geocodes each address into coordinates and keeps only successful results, storing them in a structured pandas DataFrame.

4. Converts the DataFrame into a GeoDataFrame with point geometries so the job data can be used in spatial analysis workflows.

5. Loads Czech district boundaries from the okresy.json file and ensures both datasets use the same coordinate reference system.

6. Performs a spatial join to assign each job point to a district, then aggregates job counts per district and merges results back into the map.

7. Visualizes the final dataset as a choropleth map, coloring districts by job density and adding readable labels for high-activity areas.

Step #6: Run the Script

Execute the script, and you’ll get a result like this:

Notice how most fresh software engineering openings are concentrated in districts surrounding Prague, Brno, and Ostrava. These are the three largest cities in the Czech Republic, so the result clearly makes sense!

Now, this was just a simple example, but you can use the same approach to build an interactive map or add additional features, such as calculating the average salary for each position by district, and more advanced analytics.

Value and Playbook for Other Governments

The approach presented in this blog post is highly adaptable to any public dataset with a spatial dimension, where location plays a key role in understanding patterns and inequalities. Still, a reusable country-independent playbook emerges:

1. Collect data via web scraping (or other administrative, public sources).

2. Clean and validate the data.

3. Enrich and aggregate it with contextual and geographic features.

4. Apply statistical or analytical models.

5. Present the results through maps, dashboards, or other interactive tools.

As you can tell, this pipeline isn’t domain-specific and can be reused across many different policy areas. Finally, it’s crucial that the adopted methodology remains open and clearly explained to the public. This fosters transparency, reproducibility, and trust in data-driven decision-making.

Conclusion

Here, I’ve shown how the Czech Ministry of Finance uses a web scraping service as part of a full data pipeline to turn rental listings into actionable insights for housing policy and public tools.

By accessing data collected from multiple real estate portals and processing it through cleaning, geolocation, and statistical modelling, they transform raw web data into interactive maps and rent calculators that support decision-making.

In this article, I built a similar pipeline for a different use case: a job market choropleth across Czech districts. I started from scraped Indeed listings, geocoded job locations, mapped them to districts, and visualized the results on a map.

I hope this example was useful and inspiring. If you have questions or ideas, feel free to share them in the comments below.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

You can read that episode two ways. Either the guardrails were the only thing standing between a public model and a stream of attack-ready output, or the shutdown was a tidier story than the truth. We are not going to settle that here. For our corner of the world, the lesson is simpler. A capable model, stripped of guardrails or jailbroken around them, automates the expensive part of an attacker’s job. Deobfuscation, code analysis, and figuring out how an unfamiliar system behaves under pressure. Those tasks have always gated the scraper-versus-vendor arms race because they cost human reverse-engineering hours, and that gate is now moving.

The cat-and-mouse game we cover every week runs on a hidden economy. A vendor ships an obfuscated sensor. Someone on the scraping side spends days or weeks reading minified JavaScript, renaming variables, and rebuilding the logic until the parameters make sense. The cost of that labor is what keeps most protections standing. If a model can do a real share of it on its own, the math behind the whole arms race changes.

Manus AI, and why not Claude

Manus is not a chat assistant. It is an agent that pairs a language model with a sandboxed virtual machine, and inside that machine, it runs code, fetches pages, beautifies scripts, and iterates on its own output. Meta acquired it at the end of December 2025 for around two billion dollars, though by June 2026, Chinese regulators were already forcing parts of that deal to unwind. What that architecture buys is verification. Manus not only describes what a script might do. It runs the script and checks.

That difference shows up the moment you set guardrails next to it. Ask Claude or ChatGPT to deobfuscate a live anti-bot sensor and explain which signals feed its scoring, and you tend to hit a wall. Those products keep tight cybersecurity guardrails, and the Fable shutdown is the clearest sign yet of how hard that line is enforced. We handed the same class of task to Manus, and it got to work. We have looked at agentic browsers and AI scraping assistants before, but those were aids for a human operator who still did the thinking. Here, the model does the reverse engineering.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Mapping Akamai on Net-a-Porter

We have written about Akamai more than any other anti-bot system. For the bypass-oriented background, THE LAB #30, THE LAB #85, and THE LAB #100 cover the recovery-when-nothing-works case, proxy chaining, and the browser-login-plus-HTTP hybrid. We will not repeat any of that.

What we wanted this time was not a bypass. It was a map. Akamai Bot Manager protects net-a-porter.com, and it defends in stacked layers that each fail differently:

• A TLS fingerprint check at the edge, evaluated before a single line of the client JavaScript runs.

• A sec-cpt cryptographic challenge, whose state lives in the bm_sc cookie.

• A behavioral sensor that serializes a binary telemetry payload and posts it to a collect endpoint, validating the _abck cookie.

• A client-side anti-hook layer that checks the browser environment for tampering and reports integrity separately.

The model we asked Manus to build covers the client sensor and the challenge flow, plus whatever it could observe about the TLS gate from the outside. A working end-to-end bypass was out of scope. We are not shipping code with this article, and we did not ask Manus to produce a runnable solver, even though our tests suggest we are really close to a working solution. What we’ll share here is a summary of the analysis Manus did; in the code repository, you will find the full details.

Check the TWSC YouTube Channel

What we handed the agent

The task was deliberately cold. We did not give Manus our own Akamai notes, our wiki, or any of the prior Lab articles. We pointed it at net-a-porter.com and asked it to study how Akamai's protection works, deobfuscate the client script, and explain which parameters feed into the sensor. Nothing more.

We chose this target for three reasons. It runs a real, current Akamai deployment on a high-value luxury store. The sensor was observable live, so the agent could fetch and instrument the actual script rather than reason about a stale sample. And we already understand Akamai well enough internally to judge whether the output was accurate or confident nonsense.

The reason the task went to Manus rather than a chat assistant is the guardrail gap from the previous section. Deobfuscating a production anti-bot sensor and enumerating the signals it scores is exactly the kind of request the heavily guarded models decline. Manus took it, ran the analysis in its VM, and came back with the two documents below.

What came back

The output was more detailed than we expected from a single task run. We are sharing it in two parts and deliberately not sharing it all.

The two documents below are Manus AI’s output. We reproduce them as reportage, to show what an agent derived on its own. We have not independently re-run every claim, re-derived every value, or confirmed that each detail still holds against the live site. Treat the specifics as the agent’s findings, not as TWSC-verified ground truth. The sensor version Manus reports, 746702557, was compiled on 2025-05-09, so some details may already have rotated. Where the agent surfaced operational specifics that amount to a turnkey recipe (exact module filenames, the full endpoint table, the CSS selector ids, the challenge request names), we have redacted them.

This is a schema of how Akamai Bot Protection according to Manus AI

Part one, reproduced in full, the anti-hooking and TLS summary

This first document is the safe exhibit. It explains how Akamai detects a tampered browser and how the TLS gate at the edge decides who even gets a challenge. We reproduce it verbatim.

<iframe height="0" width="0" style="display: none; visibility: hidden;" sc="true" tabindex="-1"></iframe>

Part two, reproduced with redactions, the deep technical analysis

The second document goes much further. It walks the bootstrapping sequence, the module system, the fingerprinting surface, and the telemetry transport. This is the part that starts to read like a blueprint, so we keep the architecture and the reasoning and we strip the operational specifics. Redaction markers below are ours, not Manus’s.

If you want to see the full analysis, you can find it in our GitHub repository reserved for paying users, inside the folder 108.AKAMAI-MANUS.

Read more

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

IP sourcing in the proxy industry

Two articles written by Spur Intelligence caught my attention in the past week.
It started with a webinar I’ve attended, where they explained what ISP proxies are and how proxy companies get those IPs.

The analysis is great, and I’ve understood better how the proxy industry works.
They also mentioned how NetNut gets high-quality ISP IP addresses, and described that in detail in this article.

How Proxy Providers Co-opt Entire Networks

Spur reverse-engineers how Netnut sources its ISP proxies through a partner called DiviNetworks, which installs GRE tunnels and policy-based routing directly on the border routers of real ISPs. The result is genuine ISP IP space sold as proxy inventory, with actual subscribers still living on the same addresses. The piece is worth reading for the detection detail: outbound connections sit in the TCP source port range 40,000 to 40,200, partner routers expose an rtr-<isp>.divinetworks.com passive-DNS pattern, and the Howard University case shows an entire /16 (AS919, around 17,000 exit IPs) co-opted at the network edge. They even quote DiviNetworks’ own figure of $13,208 a month for a US /16.
Full article here

Smart TV Apps and Residential Proxy SDKs

A more recent article, instead, highlighted how proxy companies get residential proxies. No surprises, at least on my side, in reading that Smart TVs are used as exit nodes for proxies. In fact, just like with mobile apps, developers can use SDKs from proxy providers to monetize their app installations. In some cases, it’s quite clear from the consent screen what’s happening under the hood.

All the industries that rely on web-scraped data (AI in primis, but not only) need residential proxies, so proxy providers, who are the gatekeepers for these tools, should be very careful about onboarding users with legitimate use cases and keeping fraudsters away. In my career, I’ve worked with almost every big name out there, and I’ve found this is the case. As scraping professionals, we should always remember that we’re guests (sometimes unwanted) on both the target website and the proxy infrastructure, so we should be as respectful as possible and not be driven by greed to collect data.
Full article here

Scraping Infrastructure

A great article by the engineering team at Browser Use shows how they’re spawning browsers cheaply and quickly.

How We Made Cloud Browsers 3x Cheaper and Faster

Browser Use walks through rebuilding its cloud so every session is its own Firecracker microVM, and the interesting twist is that they run Firecracker on plain EC2 instead of bare metal, accepting nested virtualization to get faster scale-up and lower cost.

The numbers are great: $0.02 per browser hour down from $0.06, VM cold start under 400ms, and create latency of 825ms at p50 across a 10,000-session test.

The engineering details that are making the difference are multiple, from mapping memory in 2MB pages plus a userfaultfd handler that preloads hot pages (resume-to-ready drops from 9.8s to 3.1s, roughly 91x fewer page-fault stalls) to two-phase vCPU pinning that took a 1,000-browser launch from 17% failed sessions to zero. They also make the case for running fully headless rather than headful, since their low-level Chromium fork pushes block-avoidance to 81% on their own benchmark.

Full article here

Read and share more of this

Want to flag and discover more articles like the ones above? Scraping News (still in beta) is where the community surfaces them. Sign up and submit what you find worth reading.

Want a concept map of everything covered here, cross-referenced with our own work and other sources worth your time?

The Web Scraping Club Wiki lives on GitHub, and as an interactive site. It is plain Markdown, so you can also clone it into your own Obsidian vault and read it locally.

You set it up, it works, and you forget about it. Three months later, someone asks: “Hey, did our competitor change their pricing last week?”

You check the pipeline dashboard, and everything looks fine: all green, no errors, last run successful some hours ago. Then you dig deeper and realize your scraper has been returning empty strings for weeks. The competitor did change their pricing, and you don’t know when this happened or what was on their website before.

In this article, you’ll learn how to build competitive scraping pipelines that make failure structurally (almost) impossible. Let’s get into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Why Competitive Intelligence Scrapers Break

Let’s start by understanding the most common failure modes that competitive scraping pipelines have:

• Structural drift: This is probably the most common. A competitor renames a CSS class, and your scraper starts returning empty strings. Your database fills with nulls, and nobody notices for weeks because the pipeline is still running.

• Soft blocks: These are nastier. You get valid HTTP responses, but the content is a CAPTCHA page or a bot-detection redirect. Your parser sees it as valid HTML, but it finds no data. So, it stores nothing, or, worse, it stores the CAPTCHA page’s text as if it were real data.

• Schema rot: This happens when a competitor evolves their product, but your data model doesn’t. They add a new pricing tier or split one pricing plan into three. Your scraper extracts what it can and drops the rest. Your competitive analysis is now based on an incomplete picture of their offering.

• Timezone and locale traps: This probably needs more attention than the others. The same competitor page can return different prices, currencies, or date formats depending on where the request originates. If your scraper runs from a US server but your competitor detects it and serves EU pricing, you’re tracking the wrong numbers.

What makes these failures dangerous is that you caused them yourself, unintentionally. Every defensive pattern you added to prevent crashes is exactly what turns a broken scraper into a quietly running one. No exceptions, no alerts, no red dashboards. Just weeks of nulls that nobody notices.

The rest of this article is about making that invisible failure visible again. Here’s a visual schema to get you introduced to the solutions proposed below:

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The Monitoring Layer: Your First Line of Defense

The first thing you should do to avoid such failures is to implement a monitoring layer. This is the part that the majority of scraping engineers skip. Because, you know how things go: the business needed the data yesterday, and you can’t lose time with monitoring…

But if you want your competitive scrapers to be resilient over time, you need to implement a monitoring system. A good one is made on top of four parts: output validation, structural fingerprinting, data freshness checks, and canary fields. Let’s discuss them!

How to Validate What Your Scraper Extracted

Managing HTTP status codes in your scrapers is not enough. You need to assert that the data you extracted is actually meaningful. The following example is taken from Stripe’s pricing page:

Here’s a practical validator using Pydantic that you can implement:

# In an activated virtual environment run: pip install httpx, bs4, pydantic

import httpx 
from bs4 import BeautifulSoup
from pydantic import BaseModel, field_validator, ValidationError
from typing import Optional
import re

# Define Pydantic model
class CompetitorPricingData(BaseModel):
    plan_name: str
    monthly_price: float
    currency: str
    features: list[str]

    @field_validator("monthly_price")
    @classmethod
    def price_must_be_positive(cls, v):
        if v <= 0:
            raise ValueError(f"Price must be positive, got {v}")
        return v

    @field_validator("currency")
    @classmethod
    def currency_must_be_valid(cls, v):
        valid_currencies = {"USD", "EUR", "GBP"}
        if v.upper() not in valid_currencies:
            raise ValueError(f"Unexpected currency: {v}")
        return v.upper()

    @field_validator("features")
    @classmethod
    def features_must_not_be_empty(cls, v):
        if len(v) == 0:
            raise ValueError("Features list is empty: possible extraction failure")
        return v

# Define scraper logic
def scrape_stripe_pricing() -> list[dict]:
    url = "<https://stripe.com/pricing>"
    headers = {
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/124.0.0.0 Safari/537.36"
        ),
        "Accept-Language": "en-US,en;q=0.9",
    }

    response = httpx.get(url, headers=headers, follow_redirects=True, timeout=15)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    plans = []

    # Get class name from selectors
    plan_cards = soup.select("div.HeroPricingSubcard")

    for card in plan_cards:
        container = card.select_one(".HeroPricingSubcard__container")
        if not container:
            plans.append({
                "plan_name": "",
                "monthly_price": 0.0,
                "currency": "USD",
                "features": [],
            })
            continue

        # Get all text nodes inside the container
        text_blocks = [t.get_text(strip=True) for t in container.find_all(True) if t.get_text(strip=True)]

        # Pull the first block that looks like a price
        price_text = ""
        for block in text_blocks:
            if re.search(r"[\\d]+[.,]?[\\d]*%?\\s*\\+?\\s*[€$£]?[\\d]*", block):
                price_text = block
                break

        price_match = re.search(r"[\\d]+\\.?[\\d]*", price_text.replace(",", "."))
        monthly_price = float(price_match.group()) if price_match else 0.0

        if "$" or "¢" in price_text:
            currency = "USD"
        elif "€" in price_text:
            currency = "EUR"
        elif "£" in price_text:
            currency = "GBP"
        else:
            currency = "UNKNOWN"

        # Use all text blocks as features
        features = text_blocks if text_blocks else []

        plans.append({
            "plan_name": "Standard",  # HeroPricingSubcard is the Standard card
            "monthly_price": monthly_price,
            "currency": currency,
            "features": features,
        })

    return plans

# Define validator logic
def send_alert(message: str):
    # Wire this to Slack, PagerDuty, email, etc.
    print(f"[ALERT] {message}")

def validate_scraped_data(raw_data: dict) -> Optional[CompetitorPricingData]:
    try:
        return CompetitorPricingData(**raw_data)
    except ValidationError as e:
        # Don't silently swallow this — alert immediately
        print(f"[VALIDATION FAILED] {e}")
        send_alert(f"Scraper validation failed: {e}")
        return None

if __name__ == "__main__":
    raw_plans = scrape_stripe_pricing()

    print(f"Found {len(raw_plans)} plan(s) on Stripe's pricing page.\\n")

    for raw in raw_plans:
        validated = validate_scraped_data(raw)
        if validated:
            print(f"[OK]   {validated.plan_name} — {validated.currency} {validated.monthly_price}/mo")
            print(f"       Features: {validated.features[:2]}{'...' if len(validated.features) > 2 else ''}")
        else:
            print(f"[FAIL] Raw data that broke validation: {raw}")

Here’s what this code does:

• Defines a Pydantic model: It leverages Pydantic to define the expected shape and rules of the scraped data. It enforces that the price is positive, the currency is one of USD, EUR, or GBP, and the features list is not empty. Any violation raises a ValidationError.

• scrape_stripe_pricing(): Fetches Stripe’s pricing page using the library httpx, parses the HTML with BeautifulSoup, and extracts pricing data using CSS selectors. If a container isn’t found, it appends a zeroed-out dict, which will intentionally fail validation downstream.

• validate_scraped_data(): Passes each raw scraped dict through the Pydantic model. On failure, it calls send_alert() instead of silently swallowing the error.

• send_alert(): A stub you can wire to your alerting stack (Slack, PagerDuty, email) in a production environment. Note that, for educational purposes, throughout the entire article, this is used to print alerts via the command line.

The result is the following:

Found 2 plan(s) on Stripe's pricing page.

[OK]   Standard — EUR 1.5/mo
       Features: ['1.5% + €0.25', '1.5% + €0.25']...
[OK]   Standard — EUR 2.5/mo
       Features: ['2.5% + €0.25', '2.5% + €0.25']...

If it’s not clear why this matters, here’s what your scraper “sees” when it’s pushed from another location:

The class is the same, but the data (the currency) is completely different! This is a common example of why you need validation that goes beyond a specific location.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Using DOM Fingerprints to Get Ahead of Structural Drift

This method is one of the most valuable things you can add to a competitive intelligence pipeline.

The idea is simple: periodically hash the DOM structure of the pages you’re scraping and store its value. When the hash changes, flag it for human review before your scraper breaks. Here is how you can implement it:

# In an activated virtual environment run: pip install requests

import hashlib
import httpx
from bs4 import BeautifulSoup
import requests

# Define parsing logic
def extract_structural_fingerprint(html: str, selector: str) -> str:
    """
    Extract a structural fingerprint from a specific section of the page.
    We hash the tag names and class names, NOT the content.
    This way, price changes don't trigger false alarms — only structural changes do.
    """
    soup = BeautifulSoup(html, "html.parser")
    container = soup.select_one(selector)

    if not container:
        return "CONTAINER_NOT_FOUND"

    # Build a structural signature
    structure = []
    for tag in container.find_all(True):
        classes = sorted(tag.get("class", []))
        structure.append(f"{tag.name}:{','.join(classes)}")

    fingerprint_str = "|".join(structure)
    return hashlib.md5(fingerprint_str.encode()).hexdigest()

# Compare hashes
def check_fingerprint(url: str, selector: str, stored_fingerprint: str) -> bool:
    """
    Returns True if the structure is unchanged, False if it has drifted.
    """
    response = requests.get(url, timeout=10)
    response.raise_for_status()

    current_fingerprint = extract_structural_fingerprint(response.text, selector)

    if current_fingerprint == "CONTAINER_NOT_FOUND":
        send_alert(f"[CRITICAL] Selector '{selector}' not found on {url}. Site may have been redesigned.")
        print("\\n[OK] Structure unchanged, safe to scrape.")
        return False

    if current_fingerprint != stored_fingerprint:
        send_alert(
            f"[WARNING] Structural change detected on {url} "
            f"(selector: {selector}). "
            f"Old: {stored_fingerprint[:8]}... New: {current_fingerprint[:8]}..."
        )
        print("\\n[WARNING] Structural change detected: hashes differ")
        print(f"  Old: {fingerprint[:8]}...")
        print(f"  New: {drifted_fingerprint[:8]}...")
        print("  → Scrape skipped. Human review required before next run.")
        return False

    return True

Below is a description of what this snippet does:

• extract_structural_fingerprint(): Parses the HTML, finds the target container via CSS selector, and builds a structural signature by iterating over every tag inside it. In this specific case, it records tag names and sorts class names, but deliberately ignores text content. That signature string is then hashed with MD5 and returned. If the selector finds nothing, it returns “CONTAINER_NOT_FOUND” instead of crashing.

• check_fingerprint(): Fetches the live page, computes the current fingerprint, and compares it against the stored one. Two failure cases are handled separately: the selector disappearing entirely (CONTAINER_NOT_FOUND) and the structure changing (hash mismatch). Both call send_alert() and return False: this signals to the caller to skip the scrape entirely.

• hashlib.md5(): Used to compress the structural signature into a short and comparable string. MD5 is not used here for security purposes; it’s used because it’s fast and collision-resistant enough for DOM comparison. Any change in tag names or class names produces a completely different hash.

In the case of a drifted fingerprint, the expected result is the following:

[STORED]  Fingerprint: 248363464b2da9b3814ea9a6dc5bd0df
[DRIFTED] Fingerprint: 7532ae7466c1f0fbc2c96471acd86ea7

[WARNING] Structural change detected: hashes differ
  Old: 24836346...
  New: 7532ae74...
  → Scrape skipped. Human review required before next run.

Note that this approach to DOM changes can be considered “classical” because it is fully based on finding a meaningful selector for the target page. The problem is that the DOM could change, but not the “meaningful” selector you choose. To improve it, you can target several selectors, or you can add a check on the whole body, but this would add more noise.

If you prefer trying a more “modern” approach that leverages LLMs to check on DOM changes, at The Web Scraping Club, we’ve already covered this in the article: “Beyond the DOM: A Practical Guide to Web Data Extraction with LLMs and GPT Vision”.

Using Canary Fields as a Heartbeat for Your Scraping Pipeline

A canary field is a field that you know changes frequently. This could be a “last updated” timestamp, a dynamic element, a session token in the page: you name it. You scrape it as a heartbeat, and if it stops changing, your pipeline is probably broken. Below is a snippet that checks on a canary field:

def send_alert(message: str):
    print(f"[ALERT] {message}")

def check_canary_field(
    current_value: str,
    stored_value: str,
    field_name: str,
    competitor: str,
    max_unchanged_hours: int = 48) -> None:
    if current_value == stored_value:
        send_alert(
            f"[WARNING] Canary field '{field_name}' for {competitor} "
            f"has not changed in over {max_unchanged_hours} hours. "
            f"Possible scraper failure or soft block."
        )
    else:
        print(f"[OK] Canary field '{field_name}' for {competitor} has changed. Pipeline looks healthy.")

When something changes, the expected output is the following:

[ALERT] [WARNING] Canary field 'last_updated' for <COMPETITOR NAME> has not changed in over 48 hours. 
Possible scraper failure or soft block.

Check the TWSC YouTube Channel

Building a Storage Architecture You Can Actually Trust

Many scraping engineers make the same mistake: they store only the processed output and throw away the raw HTML.

But in the case of competitive intelligence, that’s an issue that can cause you to work on weekends. Why? Because when a competitor changed their pricing 6 months ago, and you want to reconstruct the exact timeline, you need the raw data. Without it, history is gone forever.

The solution is a two-layer storage model: a raw and a processed layer. Here’s how the two relate:

             ┌─────────────────────────────────────────────────────┐
             │                     RAW LAYER                       │
             │  S3: raw/{competitor}/{page_type}/{date}.html.gz    │
             │  What's stored: full HTML, compressed, timestamped  │
             │  Why: so you can re-parse history if your logic     │
             │  changes or your scraper had a bug                  │
             └─────────────────────┬───────────────────────────────┘
                                   │ raw_s3_key (reference)
                                   ▼
             ┌─────────────────────────────────────────────────────┐
             │                  PROCESSED LAYER                    │
             │  SQLite/Postgres: pricing_history table             │
             │  What's stored: structured, queryable records       │
             │  Why: this is what your dashboards and alerts       │
             │  consume                                            │
             └─────────────────────────────────────────────────────┘

Basically, every record in the processed layer carries a raw_s3_key that points back to the exact HTML it was extracted from. That reference is what makes the whole architecture auditable.

The Raw Layer: Keep Every HTML Response

Store the full HTML or JSON response, timestamped, for every successful scrape. S3 is a cheap option to do so, and a year of raw HTML for a handful of competitor pages costs almost nothing. Below is an example you can implement in your pipeline:

# In an activated virtual environment run: pip install boto3

import boto3
import gzip
from datetime import datetime

def store_raw_response(
    html: str,
    competitor: str,
    page_type: str,
    bucket_name: str
) -> str:
    """
    Store the raw HTML response in S3 with a structured key.
    Compress it — HTML compresses extremely well (often 10:1).
    Returns the S3 key for reference in the processed layer.
    """
    s3 = boto3.client("s3")
    timestamp = datetime.utcnow().strftime("%Y/%m/%d/%H%M%S")
    s3_key = f"raw/{competitor}/{page_type}/{timestamp}.html.gz"

    compressed = gzip.compress(html.encode("utf-8"))

    s3.put_object(
        Bucket=bucket_name,
        Key=s3_key,
        Body=compressed,
        ContentEncoding="gzip",
        ContentType="text/html",
        Metadata={
            "competitor": competitor,
            "page_type": page_type,
            "scraped_at": datetime.utcnow().isoformat()
        }
    )

    return s3_key

def retrieve_raw_response(s3_key: str, bucket_name: str) -> str:
    """
    Retrieve and decompress a raw HTML response from S3.
    Useful for re-parsing historical data when your scraper logic changes.
    """
    s3 = boto3.client("s3")
    response = s3.get_object(Bucket=bucket_name, Key=s3_key)
    compressed_data = response["Body"].read()
    return gzip.decompress(compressed_data).decode("utf-8")

Notice the S3 structure: raw/{competitor}/{page_type}/{year}/{month}/{day}/{timestamp}.html.gz. This makes it trivial to list all scrapes for a specific competitor and page type within a date range, which is exactly what you need when reconstructing a pricing history.

Need help with your scraping project?

The Processed Layer: What Your Dashboards Actually Consume

The processed layer is your structured, queryable data: the thing your dashboards and alerts actually read. Every record here has two fields that make the whole architecture work:

• raw_s3_key, which links back to the exact HTML this record was extracted from.

• fingerprint, which captures the structural state of the page at scrape time.

The first one makes your data auditable. The second one tells you, later, whether the page had already changed when this record was written.

Below is a Python example to implement it:

import sqlite3
from dataclasses import dataclass
from datetime import datetime
from typing import Optional

@dataclass
class ProcessedPricingRecord:
    competitor: str
    page_type: str
    plan_name: str
    monthly_price: float
    currency: str
    scraped_at: datetime
    raw_s3_key: str  # The link back to the raw layer
    fingerprint: str  # The structural fingerprint at time of scrape

def store_processed_record(record: ProcessedPricingRecord, db_path: str) -> None:
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    cursor.execute("""
        CREATE TABLE IF NOT EXISTS pricing_history (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            competitor TEXT NOT NULL,
            page_type TEXT NOT NULL,
            plan_name TEXT NOT NULL,
            monthly_price REAL NOT NULL,
            currency TEXT NOT NULL,
            scraped_at TEXT NOT NULL,
            raw_s3_key TEXT NOT NULL,
            fingerprint TEXT NOT NULL
        )
    """)

    cursor.execute("""
        INSERT INTO pricing_history 
        (competitor, page_type, plan_name, monthly_price, currency, scraped_at, raw_s3_key, fingerprint)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?)
    """, (
        record.competitor,
        record.page_type,
        record.plan_name,
        record.monthly_price,
        record.currency,
        record.scraped_at.isoformat(),
        record.raw_s3_key,
        record.fingerprint
    ))

    conn.commit()
    conn.close()

A note on storage: this example uses SQLite, which is fine for a single scraper running on one machine. If you’re scraping multiple competitors in parallel or need concurrent writes, consider moving to Postgres (the schema remains the same).

A Concrete Example of Why the Raw Layer Saves You on Weekends

Imagine your competitor runs a Black Friday promotion in November and your scraper captures it. In January, your CEO asks: “When exactly did they drop prices, and by how much?”

With only the processed layer, you can answer that if your scraper was working correctly. But what if your scraper had a bug in November that caused it to extract prices incorrectly? With the raw layer, you can go back, fix the extraction logic, and re-parse the November HTML. Without it, that data is gone.

Here’s a utility function that makes re-parsing straightforward:

import boto3

def reparse_historical_data(
    competitor: str,
    page_type: str,
    start_date: str,
    end_date: str,
    bucket_name: str,
    new_parser_fn,
    db_path: str
) -> int:
    """
    Re-parse all raw HTML for a competitor/page_type within a date range.
    Useful when your extraction logic changes and you need to backfill.
    
    new_parser_fn: a callable that takes raw HTML and returns a list of dicts
    Returns the number of records reprocessed.
    """
    s3 = boto3.client("s3")
    prefix = f"raw/{competitor}/{page_type}/"

    paginator = s3.get_paginator("list_objects_v2")
    pages = paginator.paginate(Bucket=bucket_name, Prefix=prefix)

    reprocessed = 0

    for page in pages:
        for obj in page.get("Contents", []):
            key = obj["Key"]
            # Filter by date range based on key structure
            key_date = key.split("/")[3] + "-" + key.split("/")[4] + "-" + key.split("/")[5]
            if start_date <= key_date <= end_date:
                raw_html = retrieve_raw_response(key, bucket_name)
                parsed_records = new_parser_fn(raw_html)
                for record_data in parsed_records:
                    # Store with updated extraction, linked to same raw key
                    record = ProcessedPricingRecord(
                        **record_data,
                        raw_s3_key=key,
                        scraped_at=datetime.utcnow()
                    )
                    store_processed_record(record, db_path)
                    reprocessed += 1

    return reprocessed

Note that you’re not re-scraping anything. You’re just running your new parser against HTML you already have in storage. That’s the whole point of the raw layer.

Recovering From a Full Scraper Failure Without Losing History

Imagine this scenario: a competitor does a full site redesign, and your scraper is dead. This is not a hypothetical scenario: it happens more frequently than you can think, and most teams have no protocol for it.

Here’s a concrete recovery process.

How to Know Your Scraper Is Dead (Not Just Slow)

If you’ve implemented structural fingerprinting as discussed previously, you’ll know about the redesign before your scraper breaks. The fingerprint check will fail, you’ll get an alert, and you can investigate before storing bad data.

If you haven’t, and you’re reading this article after the fact, the first sign is usually a spike in validation failures or a sudden drop in scraped records. Consider implementing the following code:

import sqlite3

def detect_scraper_health(db_path: str, competitor: str, lookback_days: int = 7) -> dict:
    """
    Returns a health summary for a competitor's scraper over the last N days.
    Useful for spotting gradual degradation before it becomes a full failure.
    """
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    cursor.execute("""
        SELECT
            DATE(scraped_at) as scrape_date,
            COUNT(*) as total_attempts,
            SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successes,
            SUM(CASE WHEN status = 'validation_failed' THEN 1 ELSE 0 END) as validation_failures,
            SUM(CASE WHEN status = 'error' THEN 1 ELSE 0 END) as errors
        FROM scrape_log
        WHERE competitor = ?
          AND scraped_at >= DATE('now', ?)
        GROUP BY DATE(scraped_at)
        ORDER BY scrape_date DESC
    """, (competitor, f"-{lookback_days} days"))

    rows = cursor.fetchall()
    conn.close()

    return [
        {
            "date": row[0],
            "total": row[1],
            "successes": row[2],
            "validation_failures": row[3],
            "errors": row[4],
            "success_rate": round(row[2] / row[1] * 100, 1) if row[1] > 0 else 0
        }
        for row in rows
    ]

The expected result is something as follows:

Date         Total     OK   Val.Fail   Errors     Rate
--------------------------------------------------------
2026-01-15      10      0          0       10     0.0%
2026-01-14      10      2          5        3    20.0%
2026-01-13      10      5          4        1    50.0%
2026-01-12      10      8          2        0    80.0%
2026-01-11      10      9          1        0    90.0%
2026-01-10      10     10          0        0   100.0%
2026-01-09      10     10          0        0   100.0%

The degradation pattern is immediately readable. A success rate dropping from 95% to 60% over three days is probably a sign of a redesign in progress. A drop from 95% to 0% in a short time signals a full redesign, or a hard block.

Diagnosing What Actually Broke Before You Rewrite Anything

Not every structural change is a full redesign. Rewriting the major part of a scraper can cost you days if not weeks. To avoid that, you can write a script that checks what actually changed, pointing to meaningful parts of the target website:

from bs4 import BeautifulSoup

def triage_structural_change(
    url: str,
    selector: str,
    old_fingerprint: str,
    bucket_name: str,
    competitor: str,
    page_type: str) -> dict:
    """
    Diagnose the nature of a structural change.
    Returns a triage report to guide the recovery effort.
    """
    response = requests.get(url, timeout=10)
    current_html = response.text
    soup = BeautifulSoup(current_html, "html.parser")

    report = {
        "url": url,
        "http_status": response.status_code,
        "selector_found": soup.select_one(selector) is not None,
        "old_fingerprint": old_fingerprint,
        "new_fingerprint": extract_structural_fingerprint(current_html, selector),
        "page_title": soup.title.string if soup.title else "N/A",
        "recommendation": None
    }

    if not report["selector_found"]:
        report["recommendation"] = "FULL_REWRITE: primary selector is gone. Site likely redesigned."
    elif report["old_fingerprint"] != report["new_fingerprint"]:
        report["recommendation"] = "PARTIAL_UPDATE:selector exists but structure changed. Review child selectors."
    else:
        report["recommendation"] = "FALSE_ALARM: fingerprint mismatch may be transient. Re-check in 1 hour."

    # Store the current raw HTML for reference during rewrite
    store_raw_response(current_html, competitor, f"{page_type}_triage", bucket_name)

    return report
    
def print_report(report: dict) -> None:
    print(f"  URL:             {report['url']}")
    print(f"  Selector found:  {report['selector_found']}")
    print(f"  Old fingerprint: {report['old_fingerprint'][:8]}...")
    print(f"  New fingerprint: {report['new_fingerprint'][:8]}...")
    print(f"  Page title:      {report['page_title']}")
    print(f"  Recommendation:  {report['recommendation']}")

The expected output for the case of a full rewrite is the following:

	URL:             https://...
  Selector found:  False
  Old fingerprint: 4a7f92bc...
  New fingerprint: CONTAINER_NOT_FOUND
  Page title:      Pricing page
  Recommendation:  FULL_REWRITE: primary selector is gone. Site likely redesigned.

So, in this case, the script checks a specific URL of the target website. But the code can be, of course, generalized to more than just one. The same applies to the target selector.

Estimating How Much of the Gap You Can Recover From Raw Storage

Once you’ve rewritten the scraper, you have a gap in your data. If you’ve been storing raw HTML, you can partially fill it using a snippet like the following:

import boto3

def estimate_gap_coverage(
    competitor: str,
    page_type: str,
    failure_start_date: str,
    bucket_name: str) -> dict:
    """
    Estimate how much of the gap period can be recovered from raw storage.
    """
    s3 = boto3.client("s3")
    prefix = f"raw/{competitor}/{page_type}/"

    paginator = s3.get_paginator("list_objects_v2")
    pages = paginator.paginate(Bucket=bucket_name, Prefix=prefix)

    available_keys = []
    for page in pages:
        for obj in page.get("Contents", []):
            key = obj["Key"]
            key_date = "/".join(key.split("/")[3:6])
            if key_date >= failure_start_date.replace("-", "/"):
                available_keys.append(key)

    return {
        "gap_start": failure_start_date,
        "recoverable_snapshots": len(available_keys),
        "oldest_recoverable": available_keys[0] if available_keys else None,
        "newest_recoverable": available_keys[-1] if available_keys else None,
        "recommendation": (
            "Run reparse_historical_data() with your new parser to backfill."
            if available_keys else
            "No raw snapshots available for this period. Gap cannot be recovered."
        )
    }

# Print values
print(f"Gap start:              {result['gap_start']}")
print(f"Recoverable snapshots:  {result['recoverable_snapshots']}")
print(f"Oldest recoverable:     {result['oldest_recoverable']}")
print(f"Newest recoverable:     {result['newest_recoverable']}")
print(f"Recommendation:         {result['recommendation']}")

The expected result is:

Gap start:              2026-04-12
Recoverable snapshots:  3
Oldest recoverable:     <COMPETITOR_NAME>/pricing/2026/04/12/143022.html.gz
Newest recoverable:     <COMPETITOR_NAME>/pricing/2026/04/14/143022.html.gz
Recommendation:         Run reparse_historical_data() with your new parser to backfill.

As a final note on this recovery section, consider that, in practice, these three steps run in sequence:

• When something breaks, you start with detect_scraper_health() to confirm there’s actually a problem and understand how long it’s been degrading.

• Then you run triage_structural_change() to understand what broke. That tells you whether you need a full rewrite or a 10-minute fix.

• Once you’ve updated your scraper, you call estimate_gap_coverage() to see how much of the gap period you can recover from raw storage, and then reparse_historical_data() to actually backfill it. If estimate_gap_coverage() comes back with zero recoverable snapshots, that gap is gone, which is exactly why the raw layer exists in the first place.

Conclusion

Building competitive intelligence scrapers is not as easy as it seems for two reasons:

• Developing the scrapers is only half the effort.

• Keeping the scrapers honest 12 months from now probably needs even harder efforts than coding the scrapers themselves.

The layers presented in this article can make your competitive intelligence scrapers resilient through time (with the right adjustments for production environments). So, if you’re implementing this from scratch, don’t try to build everything at once. Add one piece at a time and take your time to verify everything works fine before you add the next step.

Then, you just need to analyze the data you scraped (maybe directly implementing a dashboard in Streamlit), and your colleagues in the business department can have sweet dreams: their competitive analytics are safe through time.

So, let us now: are your competitive intelligence scrapers time-resilient?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Kameleo Docker brings Kameleo’s anti-detect browser capabilities into a containerized setup, enabling production-ready automation with real fingerprinting and multi-profile isolation on Linux servers.

In this post, I’ll take a deep look at this solution and walk you through everything you need to know about it. By the end, you’ll understand what Kameleo Docker is, how its stealth browser approach works, how to set it up, and whether it’s actually worth trying.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

An Introduction to Kameleo Docker

Dig into the world of Kameleo Docker!

What Is Kameleo Docker?

Kameleo is an anti-detect browser engineered to make browser sessions look like real user devices. Instead of exposing a generic automation fingerprint, it creates realistic browser identities by spoofing hardware, browser, and environment signals such as WebGL, Canvas, fonts, screen resolution, and geolocation.

Kameleo Docker brings that same stealth stack into a self-hosted, containerized deployment model. Rather than relying on a desktop app, you can run Kameleo inside Docker on Linux or Windows servers, CI pipelines, VPSs, or Kubernetes environments.

Playwright, Puppeteer, and Selenium can connect to the container via CDP, meaning you can keep using your existing browser automation logic with minimal changes.

Further reading:

• Kameleo Docker product page

• Kameleo Docker docs

• Kameleo on Linux via Docker: What We Built, What Broke, What’s Next

Why Kameleo Docker Exists

Cloud servers, Kubernetes clusters, CI/CD pipelines, and VPS environments are overwhelmingly Linux-native, making Linux compatibility a practical requirement for automation and scraping teams.

For years, this created a problem for Kameleo users, as the solution only supported Windows and macOS. Because of that, teams wanting to run stealth browsers in production often had to rely on fragile workarounds.

Some deployed Windows virtual machines alongside Linux scraping stacks in AWS. Others used X11-over-SSH tunnels to remotely access browsers running on servers. These setups were difficult to maintain, resource-intensive, and far from ideal for scalable automation.

As explained in the product announcement blog post, Kameleo customers started to ask for a version of Kameleo that could run directly where their automation already lived.

As Barnabas Szenasi, founder and lead engineer at Kameleo, explained when I met him at Prague Crawl 2026:

“We could see from customer messages that a significant slice of our automation-first users were running Linux cloud servers and simply couldn’t use Kameleo at all... At Prague Crawl 2025, Tamas [Kameleo’s CEO] and I heard the same story from industry peers around the world: scraping pipelines were getting harder, and the need to run real browser environments instead of faking HTTP requests was growing fast.”

The Philosophy Behind the Project

From the beginning, Kameleo’s philosophy has been simple: masking quality matters more than shipping quickly.

After all, an anti-detect browser is only useful if it can convincingly behave like a real device. That’s why Kameleo relies on fingerprints sourced from real-world device traffic and continuously tested against modern anti-bot systems.

That same quality-first mindset also shaped the Docker project. According to founder and lead engineer Barnabas Szenasi, Linux support took longer than expected because the goal was never just to make Kameleo run in a container.

The objective was to reach the same masking quality users already expected on Windows and macOS. Shipping a functional but lower-fidelity Linux version would have compromised the product’s core standard.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Technical Architecture Behind Kameleo Docker

Now that you know why the project exists, let me explain how it works and the engineering behind it.

How Kameleo Docker Works

Kameleo Docker runs inside either:

• a Linux-based container (Ubuntu 22.04), or

• a Windows-based container (Windows Server Core 2022).

When you pull the image, Docker downloads the correct variant based on your container configuration (Linux containers are the default in most environments, including Docker Desktop). Regardless of the underlying platform, Kameleo exposes the same Local API for browser and profile management as the desktop app.

The process begins when you create a browser profile through the Local API. If you aren’t familiar with that concept, Kameleo profiles are reusable browser environments that bundle a complete browser fingerprint together with persistent browser state, such as cookies, browsing history, local storage, and bookmarks.

Profiles can also include user-defined settings like proxies, browser extensions, and startup preferences. Each profile is tied to a specific browser kernel and can be started, stopped, imported, or exported as needed.

Architecture Overview

Kameleo Docker separates browser execution from automation logic. The container hosts the stealth browsers, fingerprinting systems, and orchestration layer, while your automation scripts run independently on your machine, server, or orchestration platform.

At the center of the architecture is the Local API, exposed on port 5050. This API handles profile creation, fingerprint selection, browser startup, lifecycle management, and more.

Behind the API sit Kameleo’s browser kernels:

• Chroma: A Chromium-based engine.

• Junglefox: A Firefox-based engine.

These kernels are modified with engine-level masking patches and connected to Kameleo’s continuously updated fingerprint database (more on this later). Since they are exposed through the same API, you can switch between them without making any code changes.

When a profile starts, Kameleo launches a browser session with the configured fingerprint and settings. Playwright and Puppeteer can then connect to the running browser through a WebSocket endpoint via Chrome DevTools Protocol (CDP).

In other words, your automation script stays outside the Docker container. The browser behaves as if it were running locally, while execution, fingerprint masking, and browser management happen entirely inside the Docker container.

Persistent storage is handled through Docker volumes. Profile data, downloaded browser kernels, and runtime state are stored outside the container, allowing environments to be recreated without losing configuration or repeatedly downloading browser components. This makes deployments easier to scale, recover, and reproduce.

Trusted by teams running ad verification, web scraping, SERP tracking, and market research. Ethically sourced proxies, globally accessible, and fairly priced.

Get Started With DataImpulse

Core Features of Kameleo Docker

Time to explore the main features and capabilities provided by Kameleo Docker. For more information, read the official documentation.

Real Device Fingerprint Masking

Kameleo fingerprints are derived from real-world device traffic, not synthetic templates. Each profile represents a coherent combination of OS, browser, hardware signals, and behavioral characteristics.

All fingerprint surfaces are kept internally consistent, including Canvas, WebGL, audio context, screen resolution, and fonts.

Note: TLS fingerprint spoofing isn’t required, as Kameleo matches the browser kernel version precisely. The TLS stack remains the original, unmodified implementation shipped with the corresponding browser release.

The goal of the project isn’t to spoof everything, but to maintain realism across signals. That’s because overriding too many surfaces increases inconsistency risk, which detection systems can flag. For example, running a macOS fingerprint on a Windows host forces heavy compensation across system-level signals.

Proxy Integration and Geo Consistency

Each Kameleo profile can be assigned a dedicated proxy (including a rotating proxy), allowing IP-level isolation between browser identities.

Now, mismatches between IP geography and browser signals (language, timezone, WebRTC, and system locale) are a common detection vector. To address that, Kameleo provides automatic geo-location matching to align the browser’s settings with the geographic location of the selected proxy IP address.

Multi-Profile Isolation

Kameleo Docker is built around strict profile isolation, as each browser profile runs as a fully independent environment. This separation opens the door to safe multi-accounting (referred to as “account management” in Kameleo terminology). Thanks for this feature, you can operate multiple identities simultaneously without cross-contamination of session data or signals.

Linux-Specific Docker Image Features

Compared to the Windows-based container, the Linux version of Kameleo Docker includes several additional features. These include:

• Built-in VNC viewer: Allows you to monitor and interact with live browser sessions. This is especially useful for debugging automation, validating fingerprints, or troubleshooting rendering issues. You can access it through a browser on port 8080 or via native VNC clients such as RealVNC or TigerVNC on port 5900. For security reasons, it’s disabled by default.

• Browser-Based Kameleo GUI: A lightweight browser-based GUI on port 80 (reach it at http://localhost:80). Unlike the desktop app, it offers reduced functionality and is primarily intended for quick inspection, basic profile management, and monitoring.

• Optional GPU acceleration: The Linux container supports optional GPU acceleration for graphics-heavy workloads such as WebGL or canvas-intensive websites. Intel/AMD GPUs can be mounted through /dev/dri, while NVIDIA GPUs are supported through the NVIDIA Container Toolkit. When no GPU is available, Kameleo falls back to software rendering.

Getting Started With Kameleo Docker: Step-by-Step Guide

In this guided section, I’ll show you how to set up Kameleo Docker and use it for browser automation against the Scraping Course’s “JavaScript Rendering” page:

This is a sandbox environment for web scraping that simulates a real-world, JavaScript-rendered ecommerce page. It makes for a great testing target to validate the setup and see how Kameleo Docker behaves in a realistic automation scenario.

Requirements and Prerequisites

To get started with Kameleo Docker, make sure you have Docker installed and running locally. (Optionally, you need a Kameleo account if you want to run Kameleo Docker with authentication.)

For more details on supported operating systems and memory requirements, refer to the official documentation.

Since I’ll show how to use Kameleo Docker with Playwright in Python, to keep things moving, I’ll assume you already have a Python environment set up with Playwright and its dependencies installed.

To follow along with this tutorial section, I also recommend that you have:

• Basic Docker experience (running containers, mounting volumes, and using compose files)

• Familiarity with browser automation using Playwright.

Check the TWSC YouTube Channel

Step #1: Create a Kameleo Account

If you haven’t already, start by creating a Kameleo account. Fill out the sign-up form and enter the required information. Once registration is complete, a Free plan will already be activated:

Note that the Free plan is enough to use Kameleo Docker.

Important: Kameleo credentials are required for the container to authenticate successfully, download browser kernels, and start correctly.

Step #2: Start the Docker Container

With your account ready, the next step is to pull and run the Kameleo Docker image.

Remember that Kameleo ships as a multi-platform Docker image, supporting both Linux-based and Windows-based containers. To download and start the Linux container version of Kameleo, run:

docker run --platform linux/amd64 \
    --shm-size=2g \
    -p 5050:5050 \
    -e PAT='<YOUR_KAMELEO_PAT>' \
    -v kameleo-data:/data \
    kameleo/kameleo-app:latest

Note 1: If you run this command in PowerShell, replace “\” with the backtick “`” for multi-line commands.

Note 2: To launch the web GUI included in the Linux container of Kameleo Docker, add the -p 80:80 argument to your docker run command.

Here’s what matters in the above command:

• --platform linux/amd64 ensures Docker pulls the Linux-based image variant.

• --shm-size=2g is required for stable browser execution (the default Docker shared memory of 64MB is too small for browser execution).

• -v kameleo-data:/data creates a named volume that persists browser kernels and profiles across restarts.

• <YOUR_KAMELEO_PAT> is your Kameleo Personal Access Token (PAT), required to authenticate with your Kameleo account. Note that this argument is optional. You can skip the -e PAT=’<YOUR_KAMELEO_PAT>’ flag entirely if you want to try out Kameleo Docker without an account.

Below’s the output you should get:

The Kameleo Docker image should now be downloaded and launched on your system. Cool!

Step #3: Verify the Service

Once you run the image, Kameleo Docker will:

1. Start the Local API on port 5050.

2. Authenticate using your credentials.

3. Download required browser kernels (first run only).

To confirm everything is running correctly, visit the following URL in your browser:

http://localhost:5050/swagger

You should see the Swagger UI for the Kameleo Local API:

You’ll have a local Kameleo instance ready for automation with Playwright, Puppeteer, or Selenium. Great!

Step #4: Download the SDK and Create Your First Profile

Now that Kameleo Docker is running, you can interact with it through the APIs exposed at http://localhost:5050. The next step is to proceed with the usual Kameleo setup by creating a profile.

Assuming you already have a Python environment with Playwright installed,start by installing the Kameleo SDK:

pip install kameleo-local-api-client

Then, in your Playwright script, initialize the Kameleo API client and generate a browser profile:

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import CreateProfileRequest

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")
# Search for a real-world fingerprint and create a Kameleo profile based on it
fps = client.fingerprint.search_fingerprints(
    device_type="desktop",
    os_family="windows",
    browser_product="chrome",
    browser_version=">145",
)
profile = client.profile.create_profile(
    CreateProfileRequest(fingerprint_id=fps[0].id, name="twsc demo")
)

The above snippet connects the Kameleo API client from the SDK to the local Kameleo Docker APIs. It retrieves a realistic browser fingerprint from the database and creates a persistent browser profile called “twsc demo” based on it. In this case, the fingerprint profile is for a desktop Chrome browser (version >145) running on Windows.

Run the script above. If you started the Linux container of Kameleo Docker while mapping port 80 for the web GUI, then you’ll be able to see the “twsc demo” profile at http://localhost:80:

Step #5: Connect With Playwright

You can now connect Playwright to the Kameleo profile created above via CDP using the following WebSocket URL:

ws://localhost:5050/playwright/<KAMELEO_PROFILE_ID>

Achieve that by using Playwright’s connect_over_cdp() method on the target URL:

from playwright.sync_api import sync_playwright

# Kameleo profile creation...

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f"ws://localhost:5050/playwright/{profile.id}"
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Regular Playwright automation logic...

Wonderful! Playwright is attached to the browser session managed by Kameleo Docker. You can now automate it using standard Playwright APIs as if it were a regular local Chromium instance.

Step #6: Implement the Automation Logic

To achieve the scraping goal, begin by inspecting the page to study its DOM structure:

Then, apply the following Playwright logic (connected to a Kameleo profile) to automate scraping on the JavaScript-rendered page:

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f'ws://localhost:5050/playwright/{profile.id}'
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Open a new page
    context = browser.contexts[0]
    page = context.new_page()

    # Visit target site
    page.goto("https://www.scrapingcourse.com/javascript-rendering")

    # Where to store the scraped data
    products = []

    # Wait for products to render
    page.wait_for_selector(".product-item")

    # Locate all product items
    product_elements = page.locator(".product-item")

    for i in range(product_elements.count()):
        # Select the nth product
        product_element = product_elements.nth(i)

        # Extract the product data
        name = product_element.locator(".product-name").inner_text()
        price = product_element.locator(".product-price").inner_text()
        image = product_element.locator("img.product-image").get_attribute("src")
        link = product_element.locator("a.product-link").get_attribute("href")

        # Populate a product object with the scraped data
        product = {
            "name": name,
            "price": price,
            "image": image,
            "url": link
        }
        # Append it to the products list
        products.append(product)

The above snippet instructs the controlled browser to visit the target page, waits for product elements to load, then iterates through each product DOM node to extract structured fields (name, price, image, URL) and stores them in a Python list for downstream processing.

The Kameleo-powered automation script is almost complete. Only one step remains!

Step #7: Stop the Kameleo Profile

Normally, in a Playwright scenario, you would need to call browser.close() to terminate the browser session and release its resources.

In Kameleo, that’s not required. Instead, you only need to call:

client.profile.stop_profile(profile_id=profile.id)

The above line of code sends a close command to the browser via CDP. Once the browser actually stops, the Kameleo profile is terminated, too. This ensures that all resources associated with both the browser and the running profile are properly released.

Step #8: Run the Script

The final Playwright automation script, connecting via CDP to the stealth browser instance exposed by Kameleo Docker, will contain:

# pip install playwright kameleo-local-api-client

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import CreateProfileRequest
from playwright.sync_api import sync_playwright

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")
# Search for a real-world fingerprint and create a Kameleo profile based on it
fps = client.fingerprint.search_fingerprints(
    device_type="desktop",
    os_family="windows",
    browser_product="chrome",
    browser_version=">145",
)
profile = client.profile.create_profile(
    CreateProfileRequest(fingerprint_id=fps[0].id, name="twsc demo")
)

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f'ws://localhost:5050/playwright/{profile.id}'
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Open a new page
    context = browser.contexts[0]
    page = context.new_page()

    # Visit target site
    page.goto("https://www.scrapingcourse.com/javascript-rendering")

    # Where to store the scraped data
    products = []

    # Wait for products to render
    page.wait_for_selector(".product-item")

    # Locate all product items
    product_elements = page.locator(".product-item")

    for i in range(product_elements.count()):
        # Select the nth product
        product_element = product_elements.nth(i)

        # Extract the product data
        name = product_element.locator(".product-name").inner_text()
        price = product_element.locator(".product-price").inner_text()
        image = product_element.locator("img.product-image").get_attribute("src")
        link = product_element.locator("a.product-link").get_attribute("href")

        # Populate a product object with the scraped data
        product = {
            "name": name,
            "price": price,
            "image": image,
            "url": link
        }
        # Append it to the products list
        products.append(product)

    # Print the scraped products
    for product in products:
        print(product)

# Stop the Kameleo profile
client.profile.stop_profile(profile_id=profile.id)

Execute the script, and you should see output similar to this:

Notice how the script successfully scraped product data from the JavaScript-rendered page.

Once execution completes, open the Kameleo web GUI, and you’ll notice that the “twsc demo” profile is now marked as “TERMINATED”:

That doesn’t mean the profile is gone forever. Quite the opposite!

Kameleo profiles are reusable, meaning you can retrieve and start them again later to continue the browsing session with the same fingerprint and persisted state. I’ll cover exactly how to do that in a dedicated FAQ.

Mission complete! You just learned how to use Kameleo Docker for Playwright automation. With very similar logic, you can automate Puppeteer, Selenium, or any other CDP-compatible solution, in both Python and JavaScript.

Pricing Model

Kameleo Docker is included across all plans at no additional cost. You get the same core limits (concurrent browsers, number of profiles, and browser usage time) regardless of whether you run the desktop app or the containerized version. So, take a look at the official pricing page for more information.

Anti-Bot Performance Benchmarks

To test Kameleo Docker, I ran a simple script against one page protected by each major anti-bot detection system. The results are shown below:

Note: All tests were performed locally using my ISP’s residential IP address.

As shown above, in this basic experiment, Kameleo Docker achieved a 100% success rate. In contrast, Playwright consistently failed in headless mode and, in some cases, also struggled in headful mode.

Need help with your scraping project?

Final Thoughts on Kameleo Docker

What stood out when I met Barnabas at Prague Crawl 2026 (see you next year 😉) was the clear passion the team has for the project, along with their focus on quality and continuous improvement.

At the same time, when testing new products, especially technical and complex ones like Kameleo Docker, you usually stumble across bugs or unexpected behavior. I can confidently say that this wasn’t the case at all here. Everything ran smoothly from the beginning, and I didn’t encounter any issues…

On top of that, the benchmark results are promising, and I didn’t notice any significant performance lag. Thus, my honest takeaway is simple: if you’re looking for a production-ready, containerized stealth browser, or you’re simply passionate about automation and scraping, consider giving Kameleo Docker a try!

In this article, I covered what the project is about, what it offers, how it works, and how to use it. As always, remember to use Kameleo Docker only for legal and ethical web scraping and automation. Until next time!

FAQ

Is Kameleo Docker different from the desktop app?

Kameleo Docker differs from the desktop app mainly in deployment. Instead of a local GUI, it runs as a containerized service that runs on both Linux and Windows servers. Feature parity is largely preserved, including profiles, fingerprinting, and browser engines.

Can I reuse already created profiles in Kameleo Docker?

Yes! Profiles can be reused by retrieving the full list of profiles, filtering by name (or ID, if you know it), and then starting the desired profile. For example, to reuse the “twsc demo” profile, write:

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import ProfileLifetimeState

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")

# Fetch all available profiles
profiles = client.profile.list_profiles()

# Find the profile with the specific name
target_name = "twsc demo"
profile = next((p for p in profiles if p.name == target_name), None)

# Check if the profile was found
if profile:
    # Start the existing profile if it isn't already running
    if profile.status.lifetime_state != ProfileLifetimeState.RUNNING:
      client.profile.start_profile(profile.id)

Can I use Kameleo Docker in my CI/CD?

Kameleo Docker fits naturally into CI/CD pipelines by running as a disposable, reproducible container in build or test stages. You can spin up browsers on demand, run automated flows, and tear them down after execution. Configuration is typically handled via Docker Compose.

Does Kameleo Docker support proxy integration?

Yes! HTTP, HTTPS, and SOCKS proxies can be configured at profile creation time, as explained in the documentation.

Can Kameleo Docker scale to thousands of browsers?

Kameleo Docker supports horizontal scaling through standard orchestration tools. You can run multiple containers across clusters using Kubernetes or AWS ECS, each managing independent browser instances.

How Does Firefox Automation Work in Kameleo Docker?

Kameleo Docker supports Firefox-based automation through the Junglefox engine. Because Playwright cannot connect directly to Firefox-based sessions, Kameleo provides a pw-bridge helper that acts as a compatibility layer. This component translates Playwright connections into the correct browser session, allowing standard automation scripts to run unchanged while still using Firefox-based fingerprint profiles.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Shopee does not hand you the easy version. Capture a request, replay it, and the backend answers with HTTP 418 and a security error code. Every API call carries a set of anti-fraud headers, and the code that builds them is not in the Java you can read with JADX. It sits in native code, in compiled .so libraries, which is exactly where traffic interception and a Java decompiler stop being useful. Open the app in JADX and the signing method is there in name only, declared native, with its body on the far side in ARM machine code.

This is a two-part investigation into how Shopee signs its API requests and how you reproduce that signing yourself. The strategy is the one that works on most hardened apps. You locate the native security libraries, open them in a disassembler, and turn what they do back into something you control. When a library is readable crypto, you reimplement it in Python and sign offline, at any volume, with no app in the loop. When it is a bytecode virtual machine you cannot practically rewrite, you keep the app running and drive its own signer as an oracle. We chose this route because it is the one that scales. An offline signer, or an oracle you call, runs inside your scraper on a server. A rooted phone you have to babysit does not.

This first part is the foundation. We take one of Shopee’s native libraries, libshopeeaegis.so, reverse it end to end with Ghidra and rebuild it in Python. Reading the decompiled code identifies every operation as textbook crypto, and our rebuild reproduces those algorithms byte for byte. It is the readable case, the kind you win cleanly, and the clearest worked example of the method. The second part takes on the harder library, the one that computes the per-request signature, and gets past it with the oracle approach.

What you take from each part depends on your goal. If Shopee is your target, the payoff is the full picture of its request signing across both parts. If you scrape other apps, the method matters more than the marketplace. Most apps that protect their API at all push the work into a native library, and a large share of those are plain, readable crypto you can reproduce. We work it out on Aegis here, and it is the same move on the next app you open.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The tools

We use four tools, each doing one job.

androguard is a Python library for static APK analysis. We use it for fast recon. It lists the native libraries an app ships and finds which classes declare native methods. It does not give you readable source. It gives you structure you can script.

JADX decompiles Dalvik bytecode back to Java. It is how you read the managed side of the app and find the exact class and method that crosses into native code. JADX stops at the native keyword, which is the handoff point to the next tool.

Ghidra is the NSA’s open source reverse engineering framework. It disassembles a .so and decompiles it to pseudo C. It is the only tool here that can read native code, and it is the one this article leans on.

Frida injects a JavaScript engine into a running process so you can hook and call functions live. We use it to run the app under instrumentation and confirm our static reading against what the app actually does.

JADX and androguard read the managed code. Ghidra reads the native code. Frida watches the code run. The native library is the one piece only Ghidra can open, so the work centers there.

Modeling the app’s defenses

Before opening anything, it helps to name the layers, because Shopee has several and only one is our target here.

The managed layer is the Java and Kotlin code. It builds requests, attaches headers, and calls into native methods. JADX reads it.

The native layer is a set of .so libraries the app loads. Pull the arm64 split out of the APK with androguard and the security-relevant ones stand out by name. They are libshopeeaegis.so, libshpssdk.so, and libBkeBizSecurity.so, plus libjnihook.so and libshook.so. The last two are a hooking framework and an anti-hook layer, which means the app actively watches for instrumentation. That matters for Frida later.

The request-signing layer sits on top. Two okhttp interceptors, com.shopee.app.network.antifraud.b and .d (they call themselves SecurityNewSapInterceptor and SecurityNewSapPostInterceptor), attach the anti-fraud headers af-ac-enc-sz-token and x-sap-ri to API requests. The values they attach come from libshpssdk.so, the Shopee Security SDK.

We target one layer, libshopeeaegis.so, a general-purpose crypto library the app calls for specific operations. The request signer in libshpssdk.so stays out of scope. It is a bytecode virtual machine, a harder problem that we handle separately, and reproducing the af-ac headers is not the promise here. The promise is that you can take libshopeeaegis.so, understand every operation it performs, and reproduce it byte for byte in Python.

One detail decides whether that promise holds. libshopeeaegis.so loads only when the app needs it, so it is not present at idle. We watched the process maps over a minute of normal browsing and the library never appeared. The crypto we are about to reverse is a toolbox the app reaches for in certain flows, not the thing running on every request.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Getting the library and finding the door

We pulled Shopee PH 3.75.24 (com.shopee.ph) as an XAPK and unzipped it. The native libraries are not in base.apk. For a split bundle they live in config.arm64_v8a.apk. Listing the .so files with androguard and unzip -l, libshopeeaegis.so is a small one at 280 KB, which is a good sign. Small means little room for a heavy obfuscator.

androguard answers the first question, which library to open. It does not answer the second, how the app calls it. For that we go to JADX and find the class on the Java side. The library registers its native methods against com.shopee.sz.reinforce.Aegis. The class exposes a method fire, overloaded, declared native. Two of the overloads matter:

native byte[] fire(int mode, byte[] data)
native byte[] fire(int mode, byte[] data, byte[] key)

This is the door. The first argument is an integer mode. Then one or two byte arrays. The return is a byte array. JADX cannot show what fire does, because the body is in the .so. So we open the .so.

Check the TWSC YouTube Channel

Reading the library with Ghidra

We ran Ghidra 12.1.2 in headless mode. It runs without a GUI, it scripts cleanly, and it repeats exactly. The workflow is documented in our Ghidra tool skill you can use in Claude Code, just like I did for this test. In short, you import the .so, let auto analysis run, then run a script that decompiles functions to a file.

support/analyzeHeadless /tmp/proj aegis \
  -import config.arm64_v8a/lib/arm64-v8a/libshopeeaegis.so \
  -scriptPath ./scripts \
  -postScript DecompileExport.java out.c \
  -overwrite

Auto analysis finished in nine seconds and the decompiler produced 674 functions with zero failures. That number alone tells you this is not a packed or virtualized binary. A protected library fights the decompiler; this one did not.

The first useful function is JNI_OnLoad, which every JNI library runs at load time. Read its pseudo C and it looks up the class com/shopee/sz/reinforce/Aegis and calls RegisterNatives with two methods. That confirms the door from the Java side and tells us the native functions are registered dynamically rather than exported under Java_* names. Dynamic registration is a mild form of hiding, and it is exactly what Ghidra’s JNI handling and a RegisterNatives trace are for.

The C++ symbols survived. That is the break that makes this library readable. The class is Aegis, with methods named missileFire, missileCount, prism, snowDon, tugWar, and parse. There is a second class, TeslaModel, with model_3, model_a, model_b, model_c, model_e, model_s, model_x, model_y, and getNuremberg. The names are deliberately silly, a Tesla and military theme, but they are real symbols, and the structure is intact.

Follow the call chain from the registered native function. The dispatcher is Aegis::prism, a plain switch on the mode integer:

switch(param_1) {
case 0:  model_3(...)               // one input
case 1:  model_x(key, input, ...)   // keyed
case 2:  model_x(...); model_3(...) // keyed, then case 0
case 3:  model_y(...); model_3(...)
case 4:  model_e(...)
case 5:  model_a(...)
case 6:  model_b(...)
case 7:  model_s(...); model_3(...)
case 8:  model_c(...); model_3(...)
}

One native call selects one of nine operations by an integer, and some operations are a keyed primitive followed by model_3. To name each operation we read two things, the output size and the primitive body.

The output size comes from TeslaModel::getNuremberg(mode, len), which missileFire calls to size the output buffer before doing the work. It returns 16 for mode 4, 32 for mode 5, 64 for mode 6, and 20 for mode 8. Those are the digest sizes of MD5, SHA-256, SHA-512, and SHA-1. For mode 0 it returns the Base64 expansion of the input length. The size function alone half-names the table.

The bodies confirm the rest, and here the silly names get helpful, because the renamed primitives kept their original suffixes. model3_autopilot is a textbook Base64 encoder. It reads three bytes, writes four, and pads with 0x3d, which is the = character. modelx_autopilot_cbc is AES in CBC mode, recognizable because it XORs each 16 byte block with the previous ciphertext block before the round function. The hash contexts are renamed with a phantom and F theme but keep the gnulib _init_ctx / _process_bytes / _finish_ctx shape. phantom1 is SHA-1, phantom256 is SHA-256, InitF22 is SHA-512. And phantom1 as called by model_c is the HMAC form. It XORs the key with 0x36 for the inner pad and 0x5c for the outer pad over a 64 byte block, which is the HMAC construction.

Two of the keyed modes turned out not to be ciphers at all. model_s calls phantom256 with a key and a message and returns 32 bytes, so it is HMAC-SHA256. model_c calls phantom1 the same way and returns 20 bytes, so it is HMAC-SHA1. Reading the bodies kept us honest here. From the signatures alone we had guessed AES.

That gives the full table.

model_x pads with PKCS7 to a 16 byte boundary, then runs AES-CBC. The key length sets the variant, and a 16 byte key gives AES-128. model_y does the same but writes the IV in front of the ciphertext, the standard prepend-the-IV pattern, before the Base64 in mode 3.

One value is not in the file. The CBC IV is a fixed 16 byte constant the library keeps at a .bss address. .bss is zero-initialized on disk and filled at runtime, so the IV is set when the library initializes and you cannot read it statically. For the hash, HMAC, and Base64 modes that does not matter, because their output is fully determined by the input and key. For the three AES modes it means byte-identical output needs the real IV, which you read from the live process once the library loads.

As always, the code that will be used for the python reimplementation we’re showing now can be found in our GitHub repository reserved for paying users, inside the folder 107.SHOPEE-GHIDRA.

Read more

Now, since the rise of AI and LLMs, every office job has changed, with no exceptions for scraping professionals. For us, this means that surviving anti-bots and CAPTCHAs has become the first challenge when creating scraping pipelines. The second challenge has become the integration of scraping services and capabilities into AI pipelines and AI agents. The reason for this shift is simple: your boss or your clients don’t just want the data anymore. They want it in real-time, structured, and often packed with some insights.

Until some months ago, these pipelines required a huge amount of custom code (and time!). Luckily for us, the major struggle lasted until MCPs were first released.

In this article, you’ll learn what MCP is, how the Decodo MCP server works, and how to integrate it with Claude Desktop. You’ll also learn how to use it with two hands-on examples.

Let’s get into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is MCP?

The Model Context Protocol (MCP) is an open standard introduced by Anthropic that defines how AI models connect to external tools, data sources, and services. Before MCP, every integration between an AI model and an external system had to be built from scratch. MCP replaces all of the complexity behind the integration with a single, shared protocol.

Basically, the MCP protocol acts as a common language for models to connect with external tools, files, and systems. For example: do you need an AI assistant to pull a file from Google Drive, query a company database, and trigger an action in an internal app? That’s exactly the kind of job MCP is built to handle!

The practical upside is composability. Developers can mix and match several MCP servers into a single AI application without writing any integration code. So, after more than a year since its introduction in the AI industry, MCP has become the standard for integrating different services and applications into a single AI software.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

What is Decodo MCP?

The Decodo MCP Server is a web scraping layer for AI agents. It connects MCP-compatible clients to Decodo’s Web Scraping API, enabling:

• Web scraping for LLMs and AI agents without managing infrastructure: It can scrape any website, including JavaScript-heavy pages, and get real-time data without handling proxy rotation, CAPTCHA solving, or anti-bot systems. It is specifically built for RAG pipelines, AI research agents, and automation flows.

• Structured outputs for LLM workflows: It allows you to retrieve the scraped data in Markdown (LLM-ready), JSON (for structured pipelines), and screenshots (for visual context).

As of now, the Decodo MCP server exposes the following tools:

• scrape_as_markdown: Scrapes any target URL, given a target URL via prompt. Returns results in Markdown.

• screenshot: Captures a screenshot of any webpage and returns it as a PNG image.

• google_search: Scrapes Google Search for a given query, and returns parsed results.

• google_ads: Scrapes Google Ads search results.

• google_lens: Scrapes Google Lens image search results.

• google_ai_mode: Scrapes Google AI Mode (Search with AI) results.

• google_travel_hotels: Scrapes Google Travel Hotels search results.

• amazon_search: Scrapes Amazon Search for a given query, and returns parsed results.

• amazon_product: Scrapes a given Amazon Product page.

• amazon_pricing: Scrapes Amazon Product pricing information.

• amazon_sellers: Scrapes Amazon Seller information.

• amazon_bestsellers: Scrapes Amazon Bestsellers list.

• walmart_search: Scrapes Walmart Search for a given query, and returns parsed results.

• walmart_product: Scrapes Walmart Product page.

• target_search: Scrapes Target Search for a given query, and returns parsed results.

• target_product: Scrapes Target Product page.

• tiktok_post: Scrapes a TikTok post URL.

• tiktok_shop_search: Scrapes TikTok Shop Search for a given query, and returns parsed results.

• tiktok_shop_product: Scrapes TikTok Shop Product page.

• tiktok_shop_url: Scrapes TikTok Shop page by URL.

• youtube_metadata: Scrapes YouTube video metadata.

• youtube_channel: Scrapes YouTube channel videos.

• youtube_subtitles: Scrapes YouTube video subtitles.

• youtube_search: Search YouTube videos.

• reddit_post: Scrapes a specific Reddit post.

• reddit_subreddit: Scrapes Reddit subreddit results.

• reddit_user: Scrapes a Reddit user profile and their posts and comments.

• bing_search: Scrapes Bing Search results.

• chatgpt: Search and interact with ChatGPT for AI-powered responses and conversations.

• perplexity: Search and interact with Perplexity for AI-powered responses and conversations.

NOTE: Decodo is currently onboarding its MCP server onto various platforms and marketplaces. At the moment of writing this article, it can be found on the Official MCP registry, Pulse MCP, Glama AI, mcp.so, and mcpmarket.com.

How To Integrate The Decodo MCP With Claude

The prerogative of MCPs is to integrate them with LLMs. The Decodo MCP server can be integrated with several services that leverage LLMs, like Claude and Cursor. In this paragraph, you will learn how to integrate it with Claude.

Requirements

To use the Decodo MCP, your system must satisfy the following requirements:

• Claude Desktop: To integrate the Decodo MCP server with Claude, you need Claude Desktop. If you don’t have it installed yet, you can download it from their website.

• Decodo account: Create an account at dashboard.decodo.com. With a free one, you have up to 2K free requests.

• Scraping token: Get the basic authentication token. To get it, click on Web Scraping API > API playground:

  

Good. Your system is set to connect Claude Desktop with the Decodo MCP server.

Connect The Decodo MCP Server to Claude

To connect the Decodo MCP server to Claude, open Claude Desktop and click on Settings:

Then, click on Developer:

After clicking on Edit Config, the system will automatically open a folder where Claude stores all the configuration files on your local machine. Open the claude_desktop_config.json file and add the following to it:

"mcpServers": {
    "Decodo MCP Server": {
      "command": "npx",
      "args": [
        "-y",
        "@decodo/mcp-server"
      ],
      "env": {
        "SCRAPER_API_TOKEN": "<your-decodo-mcp-api-key>"
      }
    }
  } 

Replace <your-decodo-mcp-api-key> with the basic authentication token you retrieved earlier from the Decodo dashboard, and the integration is done.

Quit Claude to make the changes effective. Note that just closing the Desktop window is not sufficient. You have to quit it:

After that, when returning to Settings > Developer, you’ll see the MCP server up and running:

To be sure everything works fine, you can test it with a prompt similar to the following:

Alright, you successfully integrated the Decodo MCP server with Claude Desktop. Now it's time to test it!

Check the TWSC YouTube Channel

Running the Decodo MCP With Claude: Hands-On Examples

In this section, you will learn how to use the Decodo MCP server with two different examples:

• A basic usage taken from the documentation.

• A more advanced usage where you’ll ask Claude to retrieve some data from Amazon, returning the result in a JSON file.

Let’s get on to this!

Getting Started: Run a Google Search in Seconds

As a first and simple usage, you can test the google_search tool. The main idea behind this tool is to prompt the model with a query, and, under the hood, the MCP will use its Google search capabilities to return the result.

To just try things out, you can use the exact example reported in the Decodo MCP documentation to search for shoes on Google, reporting the top positions:

As you can see, Claude automatically tries to use the MCP server, loading its tool. As Claude is a production-ready LLM, you will be asked if you want to allow it to use the Google search from the Decodo MCP always or just for this call.

When Claude has completed its job, the result is the following:

If you want to verify that the model hasn’t hallucinated, you can search for “shoes” on Google:

Alright! You made it to your first usage example.

Need help with your scraping project?

Leveling Up: Extracting JSON-structured Data From Amazon

Having results listed in a chat can be useful for a quick overview of web research. But the actual power of MCPs is to use the capabilities of the underlying tools for specific tasks that LLMs can solve.

An example is to ask the model to retrieve some data and report the results in a JSON file. This solution provides you with data that can be further used in the second part of your pipeline—for example, for analyzing your scraped data.

For this purpose, you can use the following prompt:

Get Amazon bestsellers in electronics, extract the main info and return it in JSON format

The image below shows Amazon’s best sellers in electronics, at the time of writing this article:

Under the hood, Claude will trigger the amazon_bestsellers tool and will search the data in electronics. The chat result is the following:

Below is the (partial) JSON returned by Claude:

{
  "metadata": {
    "category": "Electronics",
    "source": "Amazon Best Sellers",
    "url": "<https://www.amazon.com/Best-Sellers/zgbs/electronics/>",
    "scraped_at": "2026-05-18",
    "total_items": 50,
    "currency": "USD"
  },
  "bestsellers": [
    {
      "rank": 1,
      "asin": "B08JHCVHTY",
      "title": "Blink Plus Plan with Monthly Auto-Renewal",
      "price": 11.99,
      "rating": 4.4,
      "ratings_count": 275779,
      "image_url": "<https://images-na.ssl-images-amazon.com/images/I/31YHGbJsldL._AC_UL300_SR300,200_.png>",
      "url": "<https://www.amazon.com/Blink-Plus-Plan-monthly-auto-renewal/dp/B08JHCVHTY>"
    },
    {
      "rank": 2,
      "asin": "B0DCH8VDXF",
      "title": "Apple EarPods Headphones with USB-C Plug",
      "price": 19,
      "rating": 4.6,
      "ratings_count": 13500,
      "image_url": "<https://images-na.ssl-images-amazon.com/images/I/513OSdW4elL._AC_UL300_SR300,200_.jpg>",
      "url": "<https://www.amazon.com/Apple-EarPods-Headphones-Built-Control/dp/B0DCH8VDXF>"
    },
    
    <Omitted for brevity>

    }
  ]
}

As you can see, the first two items correspond to the first two in the image above, taken from the Amazon best seller page. This is just to be sure the model hasn’t hallucinated.

Well done! You learned how to use the Decodo MCP server with Claude.

Conclusion

In this article, you learned what MCP is and why it has become the standard for connecting AI models to external tools and services. You also learned what the Decodo MCP server is, how to integrate it with Claude Desktop, and how to use it in practice.

So, let us know: what kind of scraping workflows are you planning to build with the Decodo MCP?

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Part of that is the cat-and-mouse game every stealth tool plays. Part of it is specific to open source. When the entire fingerprint-spoofing codebase is public, the anti-bot vendors can read it line by line and build the exact counter-signal. We made that argument in the rayobrowse review. The openness that made Camoufox popular is the same openness that let the anti-bot giants study it and catch up.

Two things changed in 2026 that make this worth a fresh look. First, the project moved. The repository at github.com/daijro/camoufox now carries a note at the top of its README:

Clover Labs is a Toronto venture studio building AI agents, listed among the project sponsors. The alpha features (per-context fingerprints, hardware spoofing) now ship first in their cloverlabs-camoufox package, and daijro’s repo became the checkpoint mirror. This is not an abandoned project; the main maintainers changed.

Second, that public repo has more than 750 forks. Open source means that when one person stops, others can pick up the work, add features on features, and keep the chase going in parallel. So the real question is not only “is Camoufox still effective”, it is “has anyone in the fork tree built something better”. This is what we tried to discover in this article.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The forks we actually tested

We pulled the fork list from the GitHub API and sorted it by recent pushes. Most of it is noise. Many forks share the exact pushed_at timestamp of the parent, which is the signature of mirror bots that never wrote a line of their own. Once you count how many commits each fork is ahead of daijro:main and read what those commits do, the field collapses to a handful. Many of the survivors only touch CI or rebrand the binary. Three of them touch the anti-detect surface for real.

Official Camoufox (github.com/daijro/camoufox) is the baseline. A custom Firefox build with a fingerprint database and stealth patches, driven through Playwright’s Juggler protocol. We covered how it hides Playwright’s own traces in THE LAB #65, so we will not repeat that here.

camoufox-reverse (github.com/WhiteNightShadow/camoufox-reverse) goes the other way. Instead of hiding harder, it adds a PropertyTracer at the SpiderMonkey engine layer that records which DOM properties a page reads. It is an instrument for watching the detector work, not a better scraper. That makes it the most useful tool in the set for understanding what we are up against.

LeooNic/camoufox (github.com/LeooNic/camoufox) is the most ambitious on paper. Its commits add content-aware canvas noise that claims to defeat a 2025 academic pixel-recovery attack, a sigma-lognormal humanized mouse engine, and RDPBrowser, an automation path that drives Firefox over the Remote Debugging Protocol instead of Juggler.

JWriter20/camoufox (github.com/JWriter20/camoufox) is the pragmatic one. Targeted stealth fixes, the headline being a closed WebRTC IP leak under a proxy on Firefox 146 (daijro issue #538), plus a real pytest suite, which none of the others ship.

Let’s start by using camoufox-reverse to discover something more about DataDome installed on Leboncoin.fr.

What DataDome reads, watched from inside the engine

Before testing who passes, we wanted to see what the detector looks at. We have explained the three detection layers before: behavioral, browser, and HTTP, in THE LAB #6. camoufox-reverse lets us watch the browser layer from below the JavaScript, which is a view we have never had in these pages.

The PropertyTracer is documented to be enabled via a config flag. We drove the macOS arm64 build directly with Playwright, set the trace config through the CAMOU_CONFIG environment variable, and pointed it at a DataDome-protected page. Our target throughout this article is leboncoin.fr, the French classifieds site, because it runs only DataDome. That isolates the signal we care about, with no second anti-bot muddying the result.

The full probe is in code/camoufox_fork_analysis/trace_datadome.py. The core of it sets the trace and lets DataDome’s script run:

config = {
    "propertyTrace": {
        "enabled": True,
        "logDir": str(LOG_DIR),
        "objects": [],            # empty = trace all covered getters
        "maxEventsPerSession": 200000,
    }
}
env = os.environ.copy()
env["CAMOU_CONFIG"] = json.dumps(config)
env["MOZ_DISABLE_CONTENT_SANDBOX"] = "1"  # required on macOS for the tracer

When sites get tough, skip the heavy lifting. Get clean, structured CSV datasets, ready for Excel, BI or your apps

Find your dataset

The tracer writes one JSON line per getter access, each shaped like {"o": "navigator", "p": "hardwareConcurrency", ...}. Loading the leboncoin homepage produced 140 engine-level reads across 30 distinct properties. Aggregated by object and property, the access pattern contains information useful for fingerprint creation:

 COUNT  PROPERTY
    14  window.outerWidth
    13  window.devicePixelRatio
    13  window.outerHeight
    13  navigator.plugins.indexedGetter
     9  navigator.hardwareConcurrency
     7  canvas.toDataURL
     6  window.innerWidth
     6  screen.rect
     4  navigator.platform
     4  navigator.userAgent
     4  webgl.getParameter
     4  canvas2d.getImageData
     3  navigator.maxTouchPoints
     2  offscreenCanvas.getContext

Check the TWSC YouTube Channel

This is happening entirely below the JavaScript layer. From the page’s point of view, nothing was instrumented, because the recording lives in the C++ getter, not in a JavaScript proxy. DataDome reads the screen geometry, the navigator core, the plugin and mime enumeration, and then it reaches for the canvas and WebGL. Both canvas.toDataURL and canvas2d.getImageData are in the list, alongside webgl.getParameter and offscreenCanvas.getContext.

That last detail is what connects this experiment to the rest of the article. The canvas readback is exactly the surface LeooNic’s content-aware noise patch sets out to protect, and the WebRTC and screen reads are where the other forks claim improvements. We now know the detector touches it all.

The homepage is the light version. The pages that hold the data are watched far more closely, and the tracer shows it. We pointed the same probe at a car listing (a leboncoin /ad/voitures/ URL). Those pages block direct connections, so this run went through a residential proxy, which is the setup we explain in the next section. The listing loaded its real content (the page title came back as “Alfa romeo Tonale 1.5 Ibrida 175ch Veloce TCT”), so we were tracing a passing ad page, not a challenge screen. The read pattern is a different animal: 584 engine-level reads across 35 properties, against 140 across 30 on the homepage.

 COUNT  PROPERTY (ad page)
   220  document.cookie.get
    47  window.innerWidth
    30  window.innerHeight
    26  navigator.plugins.indexedGetter
    26  screen.rect
    25  sessionStorage.setItem
    22  sessionStorage.getItem
    16  document.cookie.set
    12  performance.timing
     8  window.scrollY
     7  canvas.toDataURL
     6  webgl.getParameter
     4  canvas2d.getImageData
     3  navigator.globalPrivacyControl
     1  mediaDevices.enumerateDevices

The cookie reads jump from one on the homepage to 220 on the ad page. Session storage, which the homepage barely touched, is read and written dozens of times. New surfaces appear that the homepage never queried: window.scrollY for behavior, navigator.globalPrivacyControl, and mediaDevices.enumerateDevices. The canvas and WebGL reads are still there. This is the same DataDome, running a heavier script on the page that matters. It is the concrete reason the homepage passes a clean browser while the listings do not. It also tells you where to spend your effort. The protection you have to beat lives on the content pages, not the landing page.

Setting up a fair comparison

The shared virtual environment we’re creating already had camoufox 0.4.11, which fetches the Firefox 135 official build. We ran on an Apple M2 Max, so we pulled the macOS arm64 binaries for each build, signed them ad hoc (the cross-compiled bundles need it), and pointed the same launcher at each one with executable_path.

Two version details matter for fairness. JWriter20’s WebRTC fix targets a regression introduced in Firefox 146, so we did not compare it against the 135 cache. We pulled the official v146-hardware build (Firefox 146.0.1) as the baseline and JWriter20’s own 146.0.1 build as the patched version. Same Firefox, two builds. camoufox-reverse only ships at 135, which is fine because we used it only as a tracer, not as a contender.

Every test drives the binaries the way a real user would, through the camoufox launcher with proxy and geoip set, so the fingerprint database, the locale coherence, and the stealth patches are all active. The one exception is the WebRTC probe, explained below, where the page we run matters.

The WebRTC leak that JWriter20 actually fixes

JWriter20’s headline fix is a closed WebRTC IP leak under a proxy. We checked it on the official 146 build against the JWriter20 146 build, same launcher, same Bright Data proxy, geoip=True. The probe gathers ICE candidates from a STUN server and reports any IP that escapes (webrtc_leak_test.py).

A quick detour on what those candidates are, because the whole leak lives in them. WebRTC connects two peers directly, and to do that each side has to advertise every network address it could be reached on. Each address it offers is an ICE candidate. A candidate is an IP, a port, a protocol, and a type, and it reaches JavaScript as a string like this:

candidate:842163049 1 udp 1677729535 203.0.113.25 54321 typ srflx raddr 192.168.1.45 rport 54321

Two types matter here. A host candidate is an address of a local network interface, so it carries your LAN IP. A srflx (server-reflexive) candidate is the public address a STUN server reports back when the browser asks which IP it appears to come from, so it carries your real WAN IP. A page gathers all of this with no permission. It opens an RTCPeerConnection pointed at a STUN server, calls setLocalDescription, and reads each candidate as it arrives. The key is that STUN runs over UDP, and an HTTP proxy only tunnels TCP. The STUN request leaves from the real interface, the proxy never sees it, and the srflx candidate comes back with the real WAN IP even though every HTTP request went through the proxy.

The first version of our probe ran the RTCPeerConnection on about:blank and showed both builds leaking the real IP. That was our mistake, not a result. Camoufox’s content-level injection is not active on about:blank, so we were measuring an unprotected page. Moving the probe onto a real https origin changed everything:

official-146   HTTP exit IP (proxy): 189.173.138.17
               ICE candidates: 1  [srflx] ips=['203.0.113.25']   <- real WAN IP leaks

jwriter20-146  HTTP exit IP (proxy): 93.44.185.102
               ICE candidates: 0                                <- nothing leaks

Our real WAN IP is 203.0.113.25. The official build, behind a working proxy, still hands it to any page through the WebRTC reflexive candidate. The proxy exit IP rotates on each run, so the constant 203.0.113.25 in the candidate is unmistakably the real address, not the proxy.

The fix is real, and it is baked into the binary.

We unzipped both camoufox.cfg files to confirm. The official build sets only media.peerconnection.ice.no_host. JWriter20 adds default_address_only, proxy_only_if_behind_proxy, proxy_only_if_pbmode, and obfuscate_host_addresses. Behind a proxy that cannot carry UDP, those preferences make WebRTC gather no candidates at all, so there is nothing to leak. Reproduced across two runs.

If WebRTC leaks were your problem, JWriter20 solves them. Hold that thought, because it does not end where you would expect.

The canvas patch that we could read but not run

To see why this patch exists, you have to understand the small-arms race it sits within. The PropertyTracer run above caught DataDome calling toDataURL and getImageData. Those two calls are how a canvas fingerprint is taken. A script draws the same text and shapes into an off-screen canvas on every machine, reads the pixels back, and hashes them. The drawing commands are identical everywhere. The pixels are not, because the final image depends on your GPU, your graphics driver, and how your system rasterizes fonts. That hash is stable for your device and different from the next one, which is most of what a tracker wants.

The standard way to hide is to add noise. Camoufox, Brave, Firefox’s resist-fingerprinting mode, and a long tail of extensions all nudge a few pixels so the hash will not stay constant across sites. The weakness is in how that noise is generated. If it is a fixed per-session perturbation that depends only on a seed and the pixel position, it can be undone. A 2025 paper at The Web Conference, Breaking the Shield by Hoang Dai Nguyen and Phani Vadrevu, showed exactly that against eighteen extensions and five browsers. Their Pixel-Recovery attack paints a second canvas filled with a known solid color and reads it back. Because it knows what every pixel should have been, it solves for the perturbation and subtracts it from the real fingerprint canvas. Reload ten times, and the recovered fingerprint stays constant while the noised one keeps changing. That is the proof the noise was reversible all along.

Two changes defeat the attack, and the same paper points at both. Leave the flat regions alone, so a detector that paints a solid block and reads it back finds no tampering to measure. And make each perturbation depend on the pixel content rather than its position, so there is no single value to solve for and subtract. The second idea is what Brave’s Farbling does, deriving its noise from the canvas content so two different canvases are altered differently, and it is the one defense the Pixel-Recovery attack could not reverse.

LeooNic’s patch implements both moves, and it is the most interesting code in the whole fork tree. The rewritten ApplyCanvasNoise skips flat regions and only perturbs edges, and the comments name the attack directly:

// Content-aware + content-dependent canvas noise.
//   - Tier 1 known-pixel checks (DataDome, Castle): flat regions are skipped
//     because flat_score < FLAT_THRESHOLD. fillRect(R,G,B) is undisturbed.
//   - WWW'25 Pixel-Recovery Attack (Nguyen & Vadrevu): noise depends on the
//     pixel content AND its 4 neighbors, not just (seed, index).

FLAT_THRESHOLD is the cutoff that decides what counts as a flat region. The edge pixels that survive it get a content-dependent nudge of plus or minus one, small enough to stay invisible but enough to move the hash. The logic is sound on paper. We wanted to confirm it at runtime.

We built a probe that draws two solid blocks with one sharp boundary and counts perturbed pixels in the flat interior versus the edge (canvas_fingerprint_test.py).

First, we learned that canvas noise is off by default in every current build, which lines up with the CloverLabs “Disable Canvas Noise” commit. The noise only runs when canvas:seed is non-zero. With the seed forced on the official 146 build, the original algorithm shows its tell:

official-146 (original noise)
  interior flat pixels perturbed : 9105 / 18240   (~50%)
  boundary edge pixels perturbed : 98 / 192
  max edge delta (per channel)   : 1
  hash varies across sessions    : True

The stock algorithm perturbs roughly half of every pixel, flat fills included. That is precisely the behavior a known-pixel check catches, and precisely what LeooNic set out to fix. So the official baseline is the “before” picture, captured at runtime.

The “after” picture is what we were not able to collect. LeooNic ships only a Windows binary, so we ran it on a Windows cloud box. It would not launch under Playwright at all. Every attempt, headless or headful, with the stock launcher or LeooNic’s own 0.5.0 launcher installed from source, ended the same way:

console.error: "Warning: unrecognized command line flag" "-juggler-pipe"
Remote Settings startup changesets bundle could not be extracted (JSON.parse...)
JavaScript error: AsyncShutdown.sys.mjs, line 587: uncaught exception: undefined
<process did exit: exitCode=0>

The official 135 build launches and drives fine on the same box, so the machine and Playwright are healthy. The Firefox 149 build that LeooNic publishes aborts at startup before Juggler attaches.

This is not only our environment. LeooNic’s own issue #1 is titled “fix: port patches and build system to Firefox 149.0”, an open work in progress, and daijro carries issues #620 and #572 about Juggler failing to initialize in constrained environments. The build does run through LeooNic’s native RDP path, which is the whole point of their RDPBrowser, but even there we could not activate the canvas seed. The global config is ignored, and the per-context setCanvasSeed function the build exposes only at document start was never present on the page when driven over RDP.

So we report LeooNic honestly. The content-aware algorithm is real and well-reasoned in source, and the original algorithm’s weakness is confirmed at runtime on the official build. The published Firefox 149 binary is not something you can pick up and drive with the standard stack today. For a reader choosing a fork, that is the practical signal. The innovation lives in the code, not yet in a usable artifact you can run.

The block-rate test, and the result we did not expect

Given that, we could only test the jwriter fork compared to the original version. As always, the code that will be used for testing can be found in our GitHub repository reserved for paying users, inside the folder 106.CAMOUFOX.

Read more

Every product has its own price, usually formed by a simple and predictable formula. Proxy services follow a similar pricing logic. At first thought, multiplying the proxy price by the amount of bandwidth should result in a final cost, but in fact, many aspects must be taken into consideration. Bandwidth is not a clean, one-to-one reflection of useful work. What users pay for is not just the data they properly collect but also everything that happens around. And it’s about request failures, encountered blocks, timeouts, and suboptimal routing decisions.

Much of the traffic in scraping systems is consumed without usable output, and that’s where the gap between expected and actual costs starts to widen. To understand where the budget really goes, we need to research it deeper.

What is the math behind the scraping costs

Bandwidth is measurable, and proxy providers typically price it transparently. However, it’s not the key driver of the cost, it’s the result of a much more complex process. Each unit of bandwidth stands for a series of events, rather than merely one successful request. While some requests yield immediate data, many others can fail, be obstructed, time out, or require further attempts. Consequently, the actual cost structure is influenced more by system behavior than by the volume of traffic.

The real cost equals the total number of request cycles necessary to extract data multiplied by the cost of executing each request cycle. In simple words, it is defined by how many complete attempts the system must make before it gets that one response. The key detail is that only successful requests generate value. Each request cycle may have the initial request, retries after timeouts, proxy rotation, session management, and many other factors. All of them consume resources. Because of this, the real price increases not only when proxy prices rise but also when the system becomes less well-functioning. The same dataset can cost remarkably more to extract when the scraping pipeline is inefficient.

The cost amplification effect, known as the cumulative increase in resource usage caused by constant request cycles, is an issue in web scraping systems. One failed request can trigger multiple follow-up attempts. These extra cycles accumulate and raise the cost required to get one successful data point. For this reason, it’s accurate to evaluate scraping by cost per successful request. The fewer attempts needed, the lower the real cost.

Mistake #1 - Using the wrong proxy type

This mistake is not just widespread but can also cost you a lot. Not all targets need the same level of stability and anonymity, but numerous systems use a universal strategy. This results in either high costs or poor operation.

Each proxy type has its own balance of speed, cost, and detection resistance. The system will not work if that balance doesn’t align with the target website’s behavior. For example, mobile IPs are not just more expensive by default. They are highly trusted and harder to get blocked, so it’s logical to use them for challenging targets. When they are used on low-protection websites, they increase costs without improving the results. The approach that works is based on the right matching of proxy type to the task.

• Residential proxies are actively used in web scraping as they route traffic via IPs assigned by ISPs to real devices. By looking like it comes from the real user, these proxies ensure strong trust signals. Many proxy users notice better success rates when they switch to residential IPs.

• Mobile proxies direct traffic via carrier networks using IP addresses from mobile service providers. Since these IPs are shared, the traffic appears very authentic and is significantly more difficult for systems that rely on fingerprinting to identify.

• Datacenter proxies function on cloud or server-based infrastructure and use IP ranges that aren’t tied to real ISPs. The biggest advantage is speed. They are perfect for heavy automation and data collection tasks.

Mistake #2 - The retry loop problem

The goal of retry logic is to improve success rates by giving failed requests another attempt to return a valid request. This approach works when responses are consistent, and failures are occasional, not systematic. On targets with rate limits or unstable responses, many retries can lead to constant failures under the same conditions. Not all failures are the same. If you got a timeout, it’s worth retrying, but if it’s a 403 error or a block, there are other actions to try. For example, you can rotate proxies or fix headers.

Retries can turn into a loop where the system keeps sending more requests but isn’t getting better results. Instead of retrying everything the same way, treat different errors differently, and adjust your behaviour based on the response. You can rotate proxies after getting a certain status code or stop retrying when a request is blocked.

Mistake #3 - Misconfigured proxy rotation

Rotating proxies aggressively is not a solution. Changing IPs too often makes traffic look unnatural and can raise suspicion. On the flip side, not rotating enough can create another issue. Thus, there must be balance. Some websites tolerate frequent IP changes, while others expect a more stable session. Treating all targets the same way is not appropriate. It’s better to adjust rotation based on the context.

If the server expects the same user behavior over time, using sticky sessions may help. In this case, it’ll help maintain session consistency and not break the flow. For bulk data extraction, you can try rotating proxies more frequently. In this situation, there is no need to preserve session context. You can also refine your rotation if you use signal-based triggers instead of fixed rules. Rotate thoughtfully, and when the system detects specific conditions like a sudden drop in success rates or status codes, adapt your proxy to it.

Mistake #4 - Ignoring caching and duplicate requests

A notable portion of scraping traffic is often dedicated to retrieving data that has already been collected. This occurs when pipelines lack deduplication or clear definitions for data freshness. It leads to repeated requests for identical resources. This process consumes bandwidth and proxy capacity without providing new information.

To address this, implement a caching layer and deduplication logic. Responses can be cached based on a time-to-live (TTL) interval that aligns with the frequency of data updates. Request fingerprints can be used to identify duplicates before requests are sent. For structured data, storing IDs or hashes of processed items allows the system to skip previously captured content.

Mistake #5 - No cost-aware proxy routing

Many scraping systems process all requests through a single proxy type. This approach can simplify implementation, but it can still lead to ineffectiveness. Different endpoints often have distinct requirements, and a universal strategy may result in unnecessary costs.

For instance, using proxies with a high trust score for simple endpoints can be expensive, whereas lower-cost proxies used for protected pages may result in blocks and required retries. Without routing logic to adapt to these variables, systems often overpay or underperform. They can’t adapt.

An alternative is to implement cost-aware routing, which matches the proxy type to the difficulty of the task. This involves using more economical options for low-risk requests and escalating to higher-trust proxies only when necessary. By monitoring metrics such as status codes, latency, and success rates, the system can determine when to switch proxy pools. For example, a blocked request can be retried using a higher-trust proxy rather than repeating the request under the same conditions.

This approach creates a more structured pipeline that balances cost and performance by allocating resources based on the specific requirements of each request.

Understanding the real price of proxies

While “price per GB” is often cited as a standard industry metric, experienced engineers understand that it fails to capture the true economic reality of data scraping. In practice, failed requests consume bandwidth and incur costs despite yielding no usable data. These unsuccessful attempts represent a negative return on investment.

Furthermore, the expenses associated with automated retries add another hidden expense. We have to look beyond the basic per-GB rate and adopt the “Cost Per Successful Request” (CPSR). This metric provides a more accurate reflection of true operational expenses.

To calculate the cost of each valid data retrieval, use the following formula:

CPSR = price per GB / 1,000 * 1 / Success Rate

In this equation, the “success rate” is the percentage of requests that return an HTTP 200 OK status along with the intended data. Organizations can make better financial decisions if they start evaluating proxy services through the lens of CPSR.

DataImpulse is a reliable provider of residential, mobile, and datacenter proxies with non-expiring traffic and a pay-as-you-go model, meaning purchased traffic remains available until it is used. This vendor offers more than 90 million IPs in 195 countries. Teams usually choose DataImpulse for web scraping, ad verification, market research, SERP monitoring, and website testing.

Why is DataImpulse cheaper than other vendors?

The pricing structure is based on the proxy sourcing method. Many providers purchase traffic rights from ISPs and resell them, which includes an additional markup. DataImpulse sources IP addresses directly through its own application and SDKs, bypassing intermediaries to avoid extra costs. This operational model complies with all legal standards.

How to reduce your scraping costs by 30-60%

Cost efficiency in data collection is primarily achieved by minimizing inefficient requests and increasing the success rate of each attempt.

1. To optimize expenses, match proxy types to the specific requirements of the task. Using cost-effective proxies for straightforward targets while reserving higher-trust proxies for more challenging endpoints can reduce unnecessary spending.

2. Refining retry logic is also important. Failures should be addressed based on their specific status codes. Avoiding repeat requests under identical conditions prevents the waste of resources.

3. Proxy rotation should be managed strategically rather than randomly. Implementing sticky sessions and rotating based on indicators such as blocks or elevated failure rates can improve both stability and overall success rates.

4. Incorporating caching and deduplication techniques helps manage traffic. By avoiding redundant requests for data that has not changed, it is possible to decrease total request volume.

5. Implement a cost-aware proxy routing strategy. Prioritize lower-cost alternatives, escalating to premium options only when strictly necessary. This approach facilitates a more efficient resource allocation model, ensuring that infrastructure investments are directed toward the areas of greatest impact.

6. Lastly, pay attention to how your scraper interacts with websites. When a browser loads a page, it also pulls images, scripts, videos, and even fonts. Thus, lots of traffic is generated. Use HTTP requests for structured data and browser-based scraping when JS rendering is necessary.

These optimizations don’t require a comprehensive system overhaul. Incremental improvements in request efficiency can harvest significant cost reductions.

Start measuring your current CPSR baseline

At first sight, scraping costs look like a simple equation between proxy price and bandwidth. But the real drivers of cost lie deeper. As we’ve seen, unnecessary retries and poor rotation strategies contribute to a growing gap between expected and actual costs. A system with low success rates will always consume more resources.

The important shift is moving away from thinking in terms of raw pricing and toward thinking in terms of efficiency. It doesn’t always require major steps, simple adjustments are key. Better proxy selection, cost-aware routing, caching, and improved retry logic are among them. From factual proxy usage data from DataImpulse, we’ve seen that even small optimizations can noticeably reduce total costs. Every request should add value, so spending must be thoughtful and deliberate. Audit your scraping pipeline against these 5 mistakes today.

Rust is becoming increasingly popular thanks to its advantages in performance, stability, and security. But what about using it for web scraping?

In this post, I’ll show you what Rust brings to the table for web scraping, why it makes sense (and when it doesn’t), and how to build a web scraper in Rust.

Main Characteristics of Rust: Quick Overview

Rust stands out because it combines performance, safety, and control in a way few programming languages do. According to the 2025 Stack Overflow Developer Survey, 14.8% of respondents reported using Rust that year, making it the 14th most popular option.

Personally, what I find most compelling about Rust is its memory safety model. Thanks to ownership and borrowing, it avoids entire classes of bugs like memory leaks or race conditions. All of that, without needing a garbage collector!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

---

Here’s what Rust looks like in its simplest form:

fn main() {
    println!("Hello, world!");
}

Even in this minimal example, you can see Rust’s explicit structure and compile-time guarantees.

Remember: In Rust, println! is not a function. It’s a macro. The ! tells the compiler: “this is a macro invocation, not a normal function call.”

Performance is another big win. Rust is compiled and extremely fast, making it ideal for high-throughput parsing on heavy HTML pages or large volumes of pages (e.g., in an offline web scraping scenario). Concurrency is also first-class, helping you manage thousands of requests in parallel without the usual headaches.

On the flip side, Rust has a steeper learning curve. If you’re coming from Python or JavaScript, getting used to the syntax and strict compiler won’t be trivial. In my experience, the first steps can feel a bit unforgiving…

Why AI Has Made Rust a Solid Choice for Web Scraping

AI is changing the nature of software development, including web development. And, as you may already have noticed, not always in a “lighter” direction. Humans struggle to deal with long scripts and source code files, but machines don’t!

Thus, AI tends to produce very long and complex HTML with a lot of elements embedded in the same page. On top of that, AI makes it trivial to generate large amounts of content, which further increases HTML size. In addition, semantic HTML is more verbose than traditional HTML.

As a result, modern web pages are getting bigger and more complex. From a scraping perspective, this translates into slower and more resource-intensive parsing. What used to be lightweight DOM trees are now dense, deeply nested structures that require more CPU and memory to process.

This is exactly where Rust starts to make sense…

Sure, it may not be the easiest programming language, but Rust’s performance makes it compelling (and in some cases even necessary). Its low-level control and zero-cost abstractions allow Rust HTML parsers to process large documents in fractions of a second, even under high concurrency.

Independent benchmarks show Rust HTML parsers ranking among the fastest available. In particular, libraries like tl stand out for their exceptional speed and low overhead.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Best Rust Web Scraping Libraries

How to Build a Scraper in Rust: A Step-by-Step Guide

In this section, I’ll guide you through the process of building a web scraper in Rust. The target web page will be Books to Scrape’s homepage. This is a static page, which is the ideal scenario for high-speed HTML parsing in Rust.

The end goal is to scrape all the book information and export it to a CSV file. Follow the instructions below!

Prerequisites

Make sure you have:

• Rust installed locally (the article refers to Rust 1.95.0).

• Some basic familiarity with Rust syntax and constructs.

Trusted by teams running ad verification, web scraping, SERP tracking, and market research. Ethically sourced proxies, globally accessible, and fairly priced.

Get Started With DataImpulse

Step #1: Set Up a Rust Scraping Project

Create a new Rust project for web scraping with:

cargo new books_rust_scraper

This will generate a new project called books_rust_scraper containing a basic “Hello, world!” program. Move into the project folder:

cd books_rust_scraper

You should now see the following file structure:

books_rust_scraper/
├── src/
│   └── main.rs
├── target/
├── .gitignore
├── Cargo.toml
└── Cargo.lock

Focus on the src/main.rs file:

This is the entry point of your application and currently contains a simple “Hello, world!” example. Test your Rust application with:

cargo run

The command executes the src/main.rs file, so the result will be:

Hello, world!

In that file, you’ll implement your Rust web scraping logic. Great!

Step #2: Install Required Dependencies

Run these commands to install the crates (Rust libraries) needed to build a Rust web scraper:

cargo add tokio --features full
cargo add reqwest
cargo add scraper
cargo add csv

These are the core dependencies:

• tokio: Enables asynchronous execution.

• reqwest: To send HTTP requests to retrieve HTML pages.

• scraper: To parse HTML and extract data using CSS selectors.

• csv: To export the scraped data to a CSV file.

After running the commands above, your Cargo.toml file should look similar to this:

[package]
name = "books_rust_scraper"
version = "0.1.0"
edition = "2024"

[dependencies]
csv = "1.4.0"
reqwest = "0.13.3"
scraper = "0.26.0"
tl = "0.7.8"
tokio = { version = "1.52.1", features = ["full"] }

Nice! You now have all the dependencies in place to start building your Rust scraper.

Step #3: Retrieve the Target Page

Use reqwest to fetch the target page with:

use std::error::Error;
use reqwest::Client;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // Initialize the HTTP client
    let client = Client::builder()
        .build()?;

    // Retrieve the target page
    let url = "https://books.toscrape.com/";
    let response = client
    .get(url)
    .send()
    .await?;

    // Extract the HTML content from the response
    let html = response.text().await?;
  
    // Parsing logic...

    // Data export logic...

    Ok(())
}

This snippet initializes an asynchronous HTTP client using Tokio, sends a GET request to the target URL, retrieves the HTML response body, and prepares it for parsing and data extraction.

If you print html, you’ll observe:

Excellent! Get ready to apply the Rust data parsing logic.

Step #4: Implement the Parsing Logic

Before implementing the web scraping logic in Rust, study the DOM of the target page. Inspect a book HTML element in the browser:

From this structure, notice how you can select all books using the article.product_pod CSS selector. For each book element, you can retrieve:

• The title and URL from h3 a.

• The image URL from .image_container img.

• The price from .price_color.

• The rating from p.star-rating.

• The stock status from .instock.availability.

First, define a struct to store that data:

#[derive(Debug)]
struct Book {
    url: String,
    image_url: String,
    title: String,
    price: String,
    rating: String,
    in_stock: bool,
}

Next, define the parse_books() function that extracts and structures the data:

use scraper::{Html, Selector};
// ...

// ...
fn parse_books(html: &str) -> Result<Vec<Book>, Box<dyn Error>> {
    // Parse the HTML content
    let document = Html::parse_document(html);

    // Define CSS selectors for the HTML elements of interest
    let book_selector = Selector::parse("article.product_pod")?;
    let title_selector = Selector::parse("h3 a")?;
    let image_selector = Selector::parse(".image_container img")?;
    let price_selector = Selector::parse(".price_color")?;
    let rating_selector = Selector::parse("p.star-rating")?;
    let stock_selector = Selector::parse(".instock.availability")?;

    // Where to store the scraped data
    let mut books = Vec::new();

    // Iterate over each book element and extract the relevant data
    for book_el in document.select(&book_selector) {
        // Apply the parsing logic
        let title_el = book_el.select(&title_selector).next().unwrap();

        let relative_url = title_el.value().attr("href").unwrap_or("");
        let url = format!(
            "https://books.toscrape.com/catalogue/{}",
            relative_url
        );

        let image_url = book_el
            .select(&image_selector)
            .next()
            .and_then(|img| img.value().attr("src"))
            .unwrap_or("")
            .to_string();

        let image_url = format!(
            "https://books.toscrape.com/{}",
            image_url.trim_start_matches('/')
        );

        let title = title_el
            .value()
            .attr("title")
            .unwrap_or("")
            .to_string();

        let price = book_el
            .select(&price_selector)
            .next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();

        let rating = book_el
            .select(&rating_selector)
            .next()
            .and_then(|e| e.value().attr("class"))
            .unwrap_or("no rating")
            .replace("star-rating", "")
            .trim()
            .to_lowercase();

        let in_stock = book_el
            .select(&stock_selector)
            .next()
            .map(|e| {
                let text = e.text().collect::<String>();
                text.to_lowercase() == "in stock"
            })
            .unwrap_or(false);

        // Collect the scraped book data
        books.push(Book {
            title,
            price,
            rating,
            in_stock,
            image_url,
            url,
        });
    }

    Ok(books)
}

This function parses raw HTML into structured data using the scraper crate. Html::parse_document() creates a DOM-like representation of the page, while Selector::parse() defines CSS selectors for targeting elements.

document.select(&book_selector) iterates over each book container. Inside each element, .select() extracts nested elements, while .value().attr() retrieves attributes such as links and titles. The .text() method collects visible text content.

Finally, all extracted values are assembled into a Book struct, and each instance is stored in a vector for later export or processing.

Check the TWSC YouTube Channel

Step #5: Export the Scraped Data

Right now, the scraped data is returned by the parse_books() function as a vector of Book structs. Next, add a function that uses the csv crate to export that data into a CSV file:

use csv::Writer;
// ...

//...
fn write_csv(books: &[Book], file_path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let mut wtr = Writer::from_path(file_path)?;

    // Write the header row
    wtr.write_record(&[
        "url",
        "image_url",
        "title",
        "price",
        "rating",
        "in_stock",
    ])?;

    for book in books {
        wtr.write_record(&[
            &book.url,
            &book.image_url,
            &book.title,
            &book.price,
            &book.rating,
            &book.in_stock.to_string(),
        ])?;
    }

    wtr.flush()?;
    Ok(())
}

Step #6: Put It All Together

This is the final code of your Rust web scraper:

// src/main.rs

use std::error::Error;
use reqwest::Client;
use scraper::{Html, Selector};
use csv::Writer;

#[derive(Debug)]
struct Book {
    url: String,
    image_url: String,
    title: String,
    price: String,
    rating: String,
    in_stock: bool,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // Initialize the HTTP client
    let client = Client::builder()
        .build()?;

    // Retrieve the target page
    let url = "https://books.toscrape.com/";
    let response = client
    .get(url)
    .send()
    .await?;

    // Extract the HTML content from the response
    let html = response.text().await?;

    // Parse the books data from the HTML
    let books = parse_books(&html)?;

    // Export the scraped data to a CSV file
    write_csv(&books, "books.csv")?;

    Ok(())
}

fn parse_books(html: &str) -> Result<Vec<Book>, Box<dyn Error>> {
    // Parse the HTML content
    let document = Html::parse_document(html);

    // Define CSS selectors for the HTML elements of interest
    let book_selector = Selector::parse("article.product_pod")?;
    let title_selector = Selector::parse("h3 a")?;
    let image_selector = Selector::parse(".image_container img")?;
    let price_selector = Selector::parse(".price_color")?;
    let rating_selector = Selector::parse("p.star-rating")?;
    let stock_selector = Selector::parse(".instock.availability")?;

    // Where to store the scraped data
    let mut books = Vec::new();

    // Iterate over each book element and extract the relevant data
    for book_el in document.select(&book_selector) {
        // Apply the parsing logic
        let title_el = book_el.select(&title_selector).next().unwrap();

        let relative_url = title_el.value().attr("href").unwrap_or("");
        let url = format!(
            "https://books.toscrape.com/catalogue/{}",
            relative_url
        );

        let image_url = book_el
            .select(&image_selector)
            .next()
            .and_then(|img| img.value().attr("src"))
            .unwrap_or("")
            .to_string();

        let image_url = format!(
            "https://books.toscrape.com/{}",
            image_url.trim_start_matches('/')
        );

        let title = title_el
            .value()
            .attr("title")
            .unwrap_or("")
            .to_string();

        let price = book_el
            .select(&price_selector)
            .next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();

        let rating = book_el
            .select(&rating_selector)
            .next()
            .and_then(|e| e.value().attr("class"))
            .unwrap_or("no rating")
            .replace("star-rating", "")
            .trim()
            .to_lowercase();

        let in_stock = book_el
            .select(&stock_selector)
            .next()
            .map(|e| {
                let text = e.text().collect::<String>();
                text.to_lowercase() == "in stock"
            })
            .unwrap_or(false);

        // Collect the scraped book data
        books.push(Book {
            title,
            price,
            rating,
            in_stock,
            image_url,
            url,
        });
    }

    Ok(books)
}

fn write_csv(books: &[Book], file_path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let mut wtr = Writer::from_path(file_path)?;

    // Write the header row
    wtr.write_record(&[
        "url",
        "image_url",
        "title",
        "price",
        "rating",
        "in_stock",
    ])?;

    for book in books {
        wtr.write_record(&[
            &book.url,
            &book.image_url,
            &book.title,
            &book.price,
            &book.rating,
            &book.in_stock.to_string(),
        ])?;
    }

    wtr.flush()?;
    Ok(())
}

Note how all previously defined functions are now called inside main(). Et voila! In just around 150 lines of code, you’ve built an efficient web scraper in Rust.

Run your scraper with:

cargo run

After execution, a books.csv file will be created in your project folder. Open it, and you will see:

This matches exactly the data shown on the target website, but now in a structured format. Mission complete!

Browser Automation in Rust: Does It Make Sense?

First of all, it’s worth noting that the ecosystem for browser automation in Rust is quite small compared to JavaScript or Python. Also, most libraries aren’t official, but rather community-backed ports like Playwright Rust or the Selenium bindings.

Now, from a technical standpoint, browser automation happens inside the browser itself. So, Chrome, Chromium, or Firefox do most of the heavy lifting. What you define through the library’s API simply orchestrates operations like clicking, waiting for elements, and extracting data. These commands are then translated into browser actions via WebDriver, CDP, or WebDriver BiDi.

Because of that, using a systems-level language like Rust can be more of a burden than an advantage. The main strength of Rust (i.e., raw performance) doesn’t really matter here, since the controlled browser instances are the actual bottleneck, not your automation code.

That means we lose Rust’s biggest advantage while still paying its costs. On top of that, Rust’s strict compiler and steeper learning curve can slow down development speed.

To be honest, I see Rust as excellent for the parsing and data processing layer, but I wouldn’t recommend it for browser automation…

Need help with your scraping project?

Rust for Web Scraping: Final Comment

If I had to summarize my experience with Rust for web scraping, I’d say this: it really shines when you’re parsing large HTML pages at scale or handling a high number of parsing tasks in parallel.

In those scenarios, the combination of performance, memory safety, and concurrency makes a real difference. That said, I wouldn’t recommend Rust for everyday scraping tasks… The entry barrier is just too high, the learning curve too steep, and the ecosystem around scraping too small.

On top of that, finding experienced Rust developers specifically focused on web scraping, or even just translating those skills into job opportunities, can be way more challenging than in more mainstream stacks.

So my take is pretty simple: consider Rust when performance and scale truly matter. For everything else, prefer Python or JavaScript.

👍 Pros:

• Can efficiently handle thousands of requests in parallel.

• Rust HTML parsers are extremely.

• Strict compiler checks and static guarantees lead to stable scraping pipelines.

👎 Cons:

• Slower development and prototyping speed.

• Smaller ecosystem of scraping libraries compared to Python or JavaScript.

• Not a practical choice for browser automation.

Conclusion

Here, I’ve guided you through the world of web scraping in Rust. In a world dominated by AI slops, security flaws, and neglected best practices, this programming language is gaining traction thanks to its focus on efficiency and strict compilation.

As you’ve seen, Rust is excellent for CPU-intensive or memory-intensive tasks like HTML parsing and data processing. Still, it might not be ideal for browser automation or quick prototyping. You also learned how to go from zero to scraped data in CSV format by building a Rust scraper.

I hope you found this helpful and insightful. If you have any questions, feel free to share them in the comments below!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

What the 2023 LLM hype promised, and what 2026 actually shipped

The product category that took that promise most literally is the agentic browser. A real Chromium running under an LLM that reads the page, decides what to do, and clicks. Browserbase, Hyperbrowser, Skyvern, Browser Use, BrowserOS, Owl Browser, and a long tail of proxy companies rebranding their scraping browsers as “AI-powered”. The pitch in 2024 and 2025 never changed. You hand the agent a task in plain language, and the model handles the rest, including whatever defensive challenge the site throws back.

It is 2026 now, and that prediction has not played out. CAPTCHAs are still in your pipeline. So we went looking for the answer. Can an LLM actually solve a production-grade CAPTCHA like hCaptcha? We read the code where we could, checked the default configs and the docs, and probed the public surfaces of the solver services these products lean on. The picture is consistent, and it is not the one the marketing sells.

Every major agentic browser ships a CAPTCHA-solving bullet point on its site. Open the code of the open-source agents, though, and you find a different story. Almost none of them actually use an LLM to solve a CAPTCHA. They either refuse to try, or they try and fail.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

One thing to get straight before we open any code. The two families of CAPTCHA behave very differently, and the hype lands on them unevenly. The invisible ones, reCAPTCHA v3 and Cloudflare Turnstile, score your session in the background and rarely show a puzzle. A stealth-first browser on clean residential proxies usually walks past them without anyone seeing a challenge, which we covered for Turnstile in Cloudflare Turnstile: what is that and how it works? and THE LAB #73: How to Bypass Cloudflare in 2025. The visible image challenges, hCaptcha and reCAPTCHA v2, actually demand an answer. We went deep on reCAPTCHA v2 in Bypassing reCAPTCHAs With Open Source and Commercial Tools - Part 2. The one that matters in 2026 is hCaptcha, and that is where this article lives, because it is where most scrapers break.

So here is the question that drives everything below. When an agentic browser says it “solves” an hCaptcha, what does its code actually do?

Tool landscape

What’s the commercial offer today

There are four families of strategy in the commercial set. In none of them is the LLM in the agent layer doing the CAPTCHA work.

The first family is stealth-first. Residential proxies, fingerprint shaping, and request patterning lower the bot score so the visible challenge never triggers. The CAPTCHA is not solved. It is prevented from appearing. That gives you the cleanest legal posture in the set, because no automated solving is happening. ZenRows is the example.

The second family is the opposite of stealth. Instead of hiding that the traffic is automated, the vendor declares it openly and relies on a business arrangement with the CAPTCHA providers to be let through. Browserbase is explicit about this. Its Stealth Mode documentation says “through Browserbase’s partnerships with CAPTCHA providers, Browserbase can resolve challenges automatically so your sessions continue without interruption”, with solving “enabled by default for all sessions”. This is the verified-bot path, the same idea behind Cloudflare’s Web Bot Auth. A declared, allowlisted agent rather than a disguised one. No model solves a puzzle on either side. The provider recognizes the partner and waves it through. For the common challenge types, according to their documentation, this works without a fight.

The third family pairs a proprietary solver with documented third-party integrations. The vendor ships its own solving for some challenge types. For the rest, it documents how to wire in an external solver service. The external solver watches for the challenge and returns the response token through its extension or REST API. The agent then submits. Hyperbrowser and Skyvern Cloud both present this shape, a native or closed-source component plus a documented third-party path. Hyperbrowser advertises “Native Cloudflare Turnstile & CAPTCHA Solving” with “No external plugins” in its post on native CAPTCHA solving, then points to an external solver for the challenge types the native one does not cover.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

The fourth family runs an in-house solver alongside the agent. Could be a vision LLM, could be classical computer vision, could be something else. No third party is in the critical path. The vendor owns the whole stack. Bright Data, Oxylabs, and Owl Browser sit here. Owl Browser puts hard numbers on the claim: “detect and automatically solve reCAPTCHA v2 (1.2s), hCaptcha (0.8s), Turnstile (0.3s), and image CAPTCHAs.” Bright Data sells “AI-based unlocking logic” that handles “CAPTCHA solving, fingerprinting, retries, best headers, location and more” on its Web Unlocker page.

The table below maps each vendor to the mechanism its public documentation surfaces, with the source page that proves it.

Check the TWSC YouTube Channel

The open-source agents we can read

For the proprietary CAPTCHA-solving strategies we cannot see how AI is used. For the open-source ones we can. So we opened their code and read exactly how they handle CAPTCHAs. Three projects matter here.

browser-use is the most popular open-source LLM agent framework for browser automation. MIT-licensed, vendor-agnostic on the LLM side. The repo contains no LLM-driven CAPTCHA-solving logic. The one CAPTCHA-related file it ships, captcha_watchdog.py, does not solve anything either. It waits for a solver running in the BrowserUse cloud proxy and blocks the agent loop until that solver reports back. Run the library locally with your own model and no cloud proxy, and the watchdog has nothing to wait for. That makes browser-use the cleanest test of the claim that a local LLM agent solves the CAPTCHA by reading the page.

Skyvern OSS is the open core of the Skyvern Cloud product. AGPL-3.0, focused on form-filling and structured workflows, written in Python.

BrowserOS is a YC S24 open-source agent-driven browser. AGPL-3.0, 11k stars on GitHub, active development. It pairs a Chromium fork with an integrated agent runtime.

Modeling hCaptcha and reading what the open-source agents actually do

Before testing, it helps to model the target. hCaptcha embeds a widget on the host page through a script served from hcaptcha.com.

The widget renders inside an iframe whose origin is hcaptcha.com, cross-origin to the host page. When the user clicks the “I am human” checkbox, the widget decides whether to issue a challenge. If it does, a second iframe opens with the puzzle dialog. The puzzle layout varies between runs (3x3 grid, 4x3 grid, area-select with a single click on an image, bounding-box, multiple-choice prompt). When the puzzle is solved, the widget writes a response token to a hidden textarea[name="h-captcha-response"] on the host page. The host form reads that textarea on submit and posts the token along with the rest of the data. The whole solve interaction happens inside a frame the host page cannot script. The same-origin policy boundary blocks it.

That last detail decides what works and what does not. An agent driving the host page through Playwright or CDP has full control over the outer page. Inside the hcaptcha.com frame, its control is limited. A browser extension runs with cross-origin privileges. It can both observe and click inside the widget frame. That asymmetry explains most of what follows.

We started by reading three open-source agent repos to see what each one had decided to do at this boundary.

Skyvern OSS bails to the human

Skyvern’s README is candid about the scope of the OSS release: “All of the core logic powering Skyvern is available in this open source repository licensed under the AGPL-3.0 License, with the exception of anti-bot measures available in our managed cloud offering.” That single sentence puts the CAPTCHA section of every Skyvern Cloud feature page outside the repo you can read.

The repo confirms the framing. In agent_functions.py, lines 800-804, the auto_solve_captchas helper returns False unconditionally. In handler.py, lines 1149-1162, handle_solve_captcha_action does exactly one thing of substance.

async def handle_solve_captcha_action(
    action: actions.SolveCaptchaAction,
    page: Page,
    scraped_page: ScrapedPage,
    task: Task,
    step: Step,
) -> list[ActionResult]:
    LOG.warning(
        "Please solve the captcha on the page, you have 30 seconds",
        action=action,
    )
    await asyncio.sleep(30)
    return [ActionSuccess()]

Thirty seconds of asyncio.sleep and a log message asking a human to handle it. Then a success result, regardless of what the human actually did. The script-generation path in skyvern_page.py, lines 1252-1254 is even more direct. solve_captcha raises NotImplementedError. A Skyvern user opened issue #1117 asking how CAPTCHA solving was meant to work in OSS. A maintainer answered plainly: “We haven’t open sourced anything related to our captcha solver / anti-bot measures. We don’t want people abusing these things, so they must remain closed source unfortunately.”

The Skyvern OSS code does not pretend to solve CAPTCHAs. Whatever Skyvern Cloud does on top of this, the OSS release hands the problem to a human and moves on.

As always, the code that will use to bypass hCaptcha can be found in our GitHub repository reserved to paying users, inside the folder 105.HCAPTCHA.

Read more

Well, not so fast.

Open data does not automatically mean free to use, free to redistribute, or free from privacy obligations. And scraping it without thinking through the implications can land you in legal trouble, get your IP banned from a public infrastructure that was never designed to handle aggressive crawlers, or cause you to expose people’s personal information.

In this article, we will go through a complete picture of the “open data” world: what the problem actually is, how to approach it correctly, and how to implement responsible open data scrapers in Python.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

What “Open Data” Actually Means Legally, Ethically, and Practically

“Open” is one of the most overloaded words in the data world. Depending on the license, the jurisdiction, and the type of data involved, the same publicly accessible dataset can be freely redistributable, commercially restricted, privacy-sensitive, or legally off-limits entirely.

So, before anything else, let’s establish what you are actually dealing with.

What “Open-Source Dataset” Actually Means (and What It Doesn’t)

Where a dataset sits on the licensing spectrum determines everything: whether you can redistribute it, whether you can use it commercially, and whether collecting it at all exposes you to liability. Here is how the spectrum breaks down:

• CC0 (Creative Commons Zero): Essentially, it is a public domain dedication. The author waives all rights. You can scrape it, redistribute it, use it commercially, and modify it.

• CC-BY (Creative Commons Attribution): It requires you to credit the original source. This means you must clearly state where the data came from, who created it, and link back to the original when you publish or redistribute it. This is the most permissive license after CC0, and it is generally easy to comply with.

• CC-BY-SA (Share-Alike): This carries the same attribution requirement as CC-BY, but adds a condition: any derivative work you publish must carry the same license. In practice, this means you cannot fold a CC-BY-SA dataset into a proprietary product and lock it down.

• CC-BY-NC (Non-Commercial): It also requires attribution, but restricts commercial use entirely. You can use the data for research, journalism, or personal projects, but the moment money is involved, you need a separate agreement with the data owner.

• ODbL (Open Database License), used by OpenStreetMap: It requires both attribution and share-alike, specifically for databases. It is worth noting that ODbL distinguishes between the database itself and the contents. Basically, you can use individual facts freely, but any public use of the database as a whole must comply with the license terms.

And then there is the grey zone, which is where most scraping engineers actually operate: data that is publicly accessible but carries no explicit license. Common cases are government portals, academic repositories, open court records, and municipal datasets. This is a huge portion of what people call “open data”. And here is the thing that matters for scraping professionals: no license does not mean free to use. In most jurisdictions, the absence of a license means the default copyright law applies. Which means the creator reserves all rights.

So before you write a single line of scraper code, the first question is not “Can I access this?” but “Under what terms am I allowed to use what I access?”

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Where the Ethical (and Legal) Risks Hide

Once you have cleared the license question, there are still several risk areas that are easy to overlook:

• License violations: This is the most obvious one. If a dataset requires attribution and you redistribute it without crediting the source, you are in breach. If it has a non-commercial clause and you use it in a commercial product, it’s the same story. These are the kind of things that generate cease-and-desist letters.

• PII embedded in “open” datasets: This is a subtler and arguably more dangerous problem than license violation. Consider open court records: they are public by design, but they contain names, addresses, and sometimes sensitive personal details. Census microdata, even when anonymized at the aggregate level, can contain individual-level records. For example, the GitHub commit history is public, but it contains email addresses, which is personal data. So, the fact that data was made public by someone else does not strip it of its privacy implications when you collect, aggregate, and store it.

• Jurisdictional complexity: A dataset hosted on a European government portal carries GDPR obligations even if you are scraping it from the United States. The GDPR applies based on where the data subjects are located, not where the scraper is running. If you are collecting data about EU residents, you are in GDPR territory regardless of your own geography.

• The aggregation problem: This is probably one of the most underappreciated risks in the scraping industry. Individually, a dataset of names, a dataset of addresses, and a dataset of employment records might each be harmless and openly licensed. But combine them, and you have created a detailed profile of real people. This is something that privacy regulations were specifically designed to prevent.

The Infrastructure Problem: Open Data Portals Are Not Built for Scrapers

Many scraping engineers come to open data with habits built on commercial targets. That experience can be misleading, because the infrastructure behind open data portals is completely different.

When you scrape a large e-commerce website or a major social media platform, you are hitting servers that are engineered to handle millions of requests per day, backed by CDNs, load balancers, and dedicated anti-bot teams. In other words, they can take a (hard) hit.

On the other hand, a municipal open data portal, a university’s research repository, or a small NGO’s dataset hosting is an entirely different story. This means that a scraper that would barely register as noise on Amazon’s servers could genuinely degrade performance for a public data portal serving thousands of researchers.

This is why scraping open data portals aggressively is arguably more unethical than doing the same to a commercial target. You are not fighting a corporation’s anti-bot system. You are potentially taking down a public resource that other people depend on.

A Four-Step Framework for Scraping Open Datasets Without Breaking Rules or Infrastructure

Every risk outlined above has a straightforward mitigation, but only if you apply it at the right point in your workflow. The mistake most scraping engineers make is treating these as afterthoughts: checking the license after the scraper is already built, thinking about PII after the data is already stored. Let’s discuss a framework that inverts this.

License-First Workflow: Read Before You Scrape

The fix for the license problem is simple in principle, even if it requires discipline in practice: make license verification the first step of your workflow.

Most well-maintained open data portals provide license information in one of these three places: a LICENSE file in the dataset’s root directory, a metadata field in the dataset’s API response, or the dataset’s documentation page. Here is a quick reference for what the licenses described above mean for your use case:

When there is no license, the safe default is not to scrape and redistribute without seeking explicit permission from the dataset owner. A short email asking for clarification is a sign of professionalism.

Check the TWSC YouTube Channel

Prefer APIs and Bulk Downloads Over Scraping

This is a rule that experienced scraping engineers sometimes forget because they are so used to reaching for their scraper toolkit: always check for an official API or bulk download endpoint before writing a scraper.

Most serious open data portals expose REST APIs or provide direct bulk download links. Using these is better in every dimension: it is faster, more reliable, more respectful of the server, and often gives you cleaner, structured data than you would get from parsing HTML.

Your workflow should be:

1. Check the portal’s documentation for an API.

2. Check for a Sitemap or structured data endpoint (as discussed in our article on robots.txt and its implications).

3. Check for bulk download links (CSV, JSON, Parquet).

4. Only fall back to HTML scraping if none of the above exist.

Scraping should be your last resort, not your first instinct.

Responsible Scraping Behavior for Open Infrastructure

When scraping is genuinely the only option, the rules of polite scraping apply. But in the case of open data portals, you should apply a higher standard than you would on a commercial target.

As covered in “best practices for ethical web scraping”, respecting rate limits, introducing delays between requests, and using a descriptive User-Agent are baseline requirements. But for open data portals, you should go further because of their weaker infrastructure. Below are additional rules you should consider:

• Respect Crawl-delay strictly: Even if major crawlers ignore it, on underfunded infrastructure, that directive is a good signal about server capacity.

• Cache responses locally: If you need to re-run your scraper for testing or debugging, you should not be hitting the server again. Cache what you have already fetched.

• Scrape during off-peak hours: For public portals serving researchers and government users, off-peak typically means nights and weekends in the portal’s local timezone.

• Scrape only what you need: This sounds obvious, but it’s easy to over-collect data “just in case”. However, for open portals, remember that every unnecessary request is a cost imposed on a public resource that stays online on an underfunded infrastructure.

Handling PII in Open Datasets

PII stands for Personally Identifiable Information. This refers to any data that can be used, alone or in combination with other data, to identify a specific individual. Think names, email addresses, phone numbers, but also subtler things like IP addresses or device IDs.

The reality is that most well-maintained open data portals go through a review process before publication, and raw PII in open datasets is not as common as you might think. The most common cases where PII can slip through are quite specific: older government datasets published before modern privacy review processes, improperly anonymized academic research deposits, or crowdsourced datasets where contributors included personal details voluntarily.

In such specific cases, the real risk for most scraping engineers is at the aggregation level. A dataset of names, a dataset of ZIP codes, and a dataset of employment records might each be perfectly clean and openly licensed in isolation. But combine them, and you have built a detailed profile of real individuals. This is something that privacy regulations like the GDPR and CPRA were specifically designed to prevent. And once you collect, store, and process that combined data, you become responsible for it, regardless of where it originally came from.

The key principle remains the usual one: identify and handle PII at collection time. Here is a schema you can use to audit the fields that are likely to contain PII:

• Direct identifiers: names, email addresses, phone numbers, national ID numbers, passport numbers, and social security numbers. These are the clearest cases as they point to a specific individual on their own, without needing to be combined with anything else. If you see these fields in a dataset, there is no ambiguity: you are dealing with PII.

• Quasi-identifiers: dates of birth, ZIP codes, job titles, gender, ethnicity, and salary ranges. None of these identify a person on their own, but they become dangerous in combination. A classic example is aggregating just three fields—say date of birth, gender, and ZIP code: this is enough to uniquely identify a great portion of the population in a country.

• Sensitive categories under GDPR: health and medical data, political opinions, religious or philosophical beliefs, biometric data, genetic data, sexual orientation, and trade union membership. This is a legally distinct class that carries stricter obligations regardless of context. In other words, you cannot process this data based on legitimate interest alone. You need explicit consent or another specific legal basis, and the bar is significantly higher than for ordinary PII.

For each PII field, decide upfront: do you need it? If not, drop it at collection time. If you do need it, apply pseudonymization (replacing the identifier with a reversible token) or anonymization (irreversible removal or generalization) before storage.

Need help with your scraping project?

Python Implementation: Putting the Full Responsible Scraping Pipeline Into Code

Principles are only useful if they translate into implementation. Below are two concrete components you can adapt for your own pipelines:

• Checking a dataset’s license before downloading anything, using CKAN’s metadata API, with a practical fallback strategy for portals that don’t run CKAN.

• Running PII detection at collection time, using field-level schema classification, with an honest discussion of where that approach has limits.

Note that the examples below omit an API-first fetch pattern and a polite scraper skeleton, even though they are covered in the framework section above. This is because those are problems with well-known, straightforward solutions that every scraping engineer should be aware of. The idea of the following sections is to provide you with lesser-known solutions, to help you get ideas to apply to your pipelines.

Checking a Dataset’s License Programmatically

Many open data portals are built on CKAN, an open-source data management system used by governments and enterprises. CKAN exposes a REST API that includes license metadata, which makes programmatic license checking straightforward.

Here is how to query a CKAN-based portal and extract license information before proceeding:

import requests

def check_dataset_license(portal_base_url: str, dataset_id: str) -> dict:
    """
    Queries a CKAN portal API to retrieve license information
    for a given dataset before any scraping begins.
    """
    api_url = f"{portal_base_url}/api/3/action/package_show"
    params = {"id": dataset_id}

    response = requests.get(api_url, params=params, timeout=10)
    response.raise_for_status()

    data = response.json()
    result = data.get("result", {})

    license_info = {
        "dataset_name": result.get("title", "Unknown"),
        "license_id": result.get("license_id", "Not specified"),
        "license_title": result.get("license_title", "Not specified"),
        "license_url": result.get("license_url", "Not specified"),
    }

    return license_info

# Example: querying the UK government's open data portal
portal = "<https://data.gov.uk>"
dataset = "road-accidents-safety-data"

license_info = check_dataset_license(portal, dataset)

print(f"Dataset: {license_info['dataset_name']}")
print(f"License: {license_info['license_title']}")
print(f"License ID: {license_info['license_id']}")
print(f"License URL: {license_info['license_url']}")

Which outputs the following:

Dataset: Road Safety Data
License: UK Open Government Licence (OGL)
License ID: uk-ogl
License URL: <https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/>

With this information in hand, you can make an informed decision before a single byte of dataset content is downloaded. Specifically, you can directly check the government license page. The image below partially shows the license page:

But what if the portal you need to scrape doesn’t run CKAN? Not all open data portals do… Socrata (used by many US city and state governments), DKAN, and custom-built portals each have different or no metadata APIs. In those cases, your fallback options are the following:

• Check for a LICENSE or METADATA file in the dataset’s root directory or bulk download package. Many portals include one.

• Look for a <link rel=”license”> tag in the dataset’s HTML page, which some portals emit as structured metadata.

• Check the portal’s documentation or “About” page, where license terms are often stated globally for all datasets.

If none of the above yield a clear answer, treat the license as unknown and do not redistribute without seeking explicit written permission from the dataset owner. A short email asking for clarification is a professional move.

PII Detection at Scrape Time

In this case, the approach depends heavily on what you actually know about the data you need to scrape. Two situations you will encounter in practice, each calling for a different strategy:

• You know the schema: If you are retrieving structured data, field-level detection is the right approach. You know which fields are likely to carry PII, so you can target them directly. This is faster, more precise, and produces far fewer false positives than running a general NER model over free text.

• You have no schema: For unstructured data, NER-based detection is a reasonable starting point, but go in with realistic expectations. A common solution is using spaCy’s en_core_web_sm, which is a small model trained on news text, so don’t expect it to do miracles for you. Another approach, which can give way better results, is using LLMs to give a structure to unstructured text.

For the structured case, here is a field-level PII detection pipeline:

import re
import hashlib
from dataclasses import dataclass, field
from typing import Any

# Fields that are unambiguously PII on their own
DIRECT_IDENTIFIER_FIELDS = {
    "name", "full_name", "first_name", "last_name",
    "email", "email_address",
    "phone", "phone_number", "mobile",
    "ssn", "national_id", "passport_number",
    "ip_address", "device_id"
}

# Fields that are not PII alone but dangerous in combination
QUASI_IDENTIFIER_FIELDS = {
    "date_of_birth", "dob", "birth_date",
    "zip_code", "postcode", "zip",
    "gender", "sex",
    "job_title", "occupation",
    "salary", "income",
    "ethnicity", "race"
}

# Regex patterns for validating suspected PII values at the content level
EMAIL_PATTERN = re.compile(r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+")
PHONE_PATTERN = re.compile(r"\\b(\\+?\\d[\\d\\s\\-().]{7,}\\d)\\b")

@dataclass
class FieldAudit:
    field_name: str
    classification: str    # "direct", "quasi", or "clean"
    original_value: Any
    processed_value: Any   # pseudonymized, generalized, or original
    action_taken: str       # "pseudonymized", "generalized", "dropped", "kept"

def pseudonymize(value: Any) -> str:
    """
    Replaces a PII value with a consistent, reversible token.
    Using a hash means the same value always produces the same token,
    which preserves referential integrity across records (e.g., you can
    still count unique users without knowing who they are).
    In production, use an HMAC with a secret key instead of plain SHA-256.
    """
    return hashlib.sha256(str(value).encode()).hexdigest()[:16]

def generalize_date(value: str) -> str:
    """
    Reduces a full date of birth to a birth year only.
    A simple but effective generalization for quasi-identifiers.
    """
    # Handles common formats: YYYY-MM-DD, DD/MM/YYYY, MM/DD/YYYY
    match = re.search(r"\\b(19|20)\\d{2}\\b", str(value))
    return match.group(0) if match else "UNKNOWN_YEAR"

def audit_record(record: dict) -> tuple[dict, list[FieldAudit]]:
    """
    Processes a single structured record field by field.
    Returns a cleaned record and a full audit trail of what was done to each field.

    Strategy:
    - Direct identifiers: pseudonymize (preserve referential integrity)
    - Quasi-identifiers: generalize where possible, pseudonymize otherwise
    - Everything else: pass through unchanged
    """
    clean_record = {}
    audit_trail = []

    for field_name, value in record.items():
        normalized = field_name.lower().strip()

        if normalized in DIRECT_IDENTIFIER_FIELDS:
            processed = pseudonymize(value)
            audit_trail.append(FieldAudit(
                field_name=field_name,
                classification="direct",
                original_value=value,
                processed_value=processed,
                action_taken="pseudonymized"
            ))
            clean_record[field_name] = processed

        elif normalized in QUASI_IDENTIFIER_FIELDS:
            # Apply field-specific generalization where we can
            if normalized in {"date_of_birth", "dob", "birth_date"}:
                processed = generalize_date(value)
                action = "generalized"
            else:
                # For other quasi-identifiers, pseudonymize as a safe default
                processed = pseudonymize(value)
                action = "pseudonymized"

            audit_trail.append(FieldAudit(
                field_name=field_name,
                classification="quasi",
                original_value=value,
                processed_value=processed,
                action_taken=action
            ))
            clean_record[field_name] = processed

        else:
            # Field is not in either PII list — pass through, but still
            # run a regex check on string values as a safety net
            if isinstance(value, str):
                if EMAIL_PATTERN.search(value) or PHONE_PATTERN.search(value):
                    # Unexpected PII in a non-PII field: flag it and pseudonymize
                    processed = pseudonymize(value)
                    audit_trail.append(FieldAudit(
                        field_name=field_name,
                        classification="direct",
                        original_value=value,
                        processed_value=processed,
                        action_taken="pseudonymized (unexpected PII in non-PII field)"
                    ))
                    clean_record[field_name] = processed
                    continue

            audit_trail.append(FieldAudit(
                field_name=field_name,
                classification="clean",
                original_value=value,
                processed_value=value,
                action_taken="kept"
            ))
            clean_record[field_name] = value

    return clean_record, audit_trail

def process_records(records: list[dict]) -> list[dict]:
    """
    Runs field-level PII detection and handling across a list of records.
    Prints an audit summary for any record where PII was found.
    """
    clean_records = []

    for i, record in enumerate(records):
        clean_record, audit_trail = audit_record(record)
        pii_fields = [a for a in audit_trail if a.classification != "clean"]

        if pii_fields:
            print(f"Record {i}: PII detected and handled in {len(pii_fields)} field(s):")
            for audit in pii_fields:
                print(f"  [{audit.classification.upper()}] {audit.field_name} "
                      f"→ {audit.action_taken}")

        clean_records.append(clean_record)

    return clean_records

# Example: a batch of records from a scraped open dataset
records = [
    {
        "record_id": "A001",
        "name": "Jane Doe",
        "date_of_birth": "1985-03-22",
        "zip_code": "SW1A 1AA",
        "incident_type": "Road accident",
        "severity": "Slight"
    },
    {
        "record_id": "A002",
        "name": "John Smith",
        "date_of_birth": "1973-11-04",
        "zip_code": "EC1A 1BB",
        "incident_type": "Road accident",
        "severity": "Serious",
        # An email that slipped into a free-text notes field
        "notes": "Witness contact: witness@example.com"
    }
]

clean = process_records(records)

The output is the following

Record 0: PII detected and handled in 3 field(s):
  [DIRECT] name → pseudonymized
  [QUASI] date_of_birth → generalized
  [QUASI] zip_code → pseudonymized
Record 1: PII detected and handled in 4 field(s):
  [DIRECT] name → pseudonymized
  [QUASI] date_of_birth → generalized
  [QUASI] zip_code → pseudonymized
  [DIRECT] notes → pseudonymized (unexpected PII in non-PII field)

A few things worth calling out in this implementation:

• Pseudonymization preserves referential integrity: Because the same input always produces the same hash token, you can still count unique individuals, join records, or track entities across datasets, without storing the raw PII. In production, replace the plain SHA-256 with an HMAC keyed on a secret, so tokens cannot be reversed by someone who also has access to the hashing algorithm.

• The regex safety net on non-PII fields: This catches the common real-world case where PII slips into a free-text or notes field that your schema classification didn’t anticipate. It is not foolproof, but it catches the obvious cases.

• The audit trail is intentional: Every field-level decision is logged. If you are ever asked to demonstrate that your collection process handled PII responsibly, you have a record of exactly what was done to each field in each record.

Conclusion

Open data is a shared resource, and how you interact with it says something about you as a professional. In this article, you learned what “open” means in the context of data scraping and how you should treat it if you want to be an ethical scraper.

So, let us know: Did we miss something? What’s your approach to handling open datasets in your scraping projects? Let’s discuss in the comments.

No wonder nearly half a trillion dollars are spent every year by financial firms on technology. Now, you probably don’t have that kind of money in the first place (and if you do, you don’t need to invest much anyway), but you might still want to collect financial data for personal use, research, academic projects, backtesting, or even just for selling it to industry giants.

No matter what you want to do with scraped financial data, there are a few pivotal tips to understand before embarking on this journey, which is exactly what I will explain here!

In this blog post, I will show why web scraping and finance are a match made in heaven and cover everything you need to know about retrieving both historical and real-time financial data from the web.

Web Scraping + Finance: A Happy Marriage

Before diving into web scraping for finance, let me explain why this is such a powerful approach and the advantages you can gain from it.

Finance Runs on (Web) Data

If there’s one thing that’s become clear over the past decade, it’s this: finance runs on data!

Financial institutions process massive volumes of market, customer, and transactional data every single day. In finance, data powers everything, from investment strategies to risk management. And the stakes are high, as bad data alone costs organizations an average of $12.9 million per year!

Data drives real-time decision-making, predictive modeling, and scenario planning. Finance teams feed that data into pipelines built around statistical analysis, machine learning, and AI to identify patterns, forecast market movements, and manage uncertainty in increasingly complex environments.

Now, here’s the central question we, web scraping enthusiasts, are all interested in: where does most of that data actually come from? A big portion of it comes from the web (not that surprising, uh?).

I’m talking about news sites, financial portals, company pages, official exchange websites, regulatory filings, institutional reports, and more. The web is essentially the largest and most dynamic data source available for financial purposes.

That’s exactly why web scraping in finance isn’t just useful. It’s foundational!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Benefits of a Data-Driven Approach in Finance

Keep in mind that it’s not just big corporations or financial firms that benefit from data. Even individual retail investors can leverage financial data scraping to gain an edge. In particular, the main advantages include:

• Informed decisions: Access to accurate historical data supports smarter investment decisions, while real-time data enables more solid trading choices.

• Market trend insights: Spot patterns and emerging trends before the wider market does.

• Risk management: Identify potential risks early and adjust strategies proactively.

• Portfolio optimization: Fine-tune asset allocation based on backtesting and up-to-date market and company data.

• Efficiency and speed: Automate data collection, reducing time spent on manual research.

I mean, financial firms wouldn’t be spending over $495 billion a year (yeah, you read that right!) on technology (mostly built around collecting, processing, and leveraging data) if it didn’t give them a real edge!

Getting vs Selling Financial Web Data: High-Level Overview

There’s no doubt that financial firms invest billions into data. But what about you, as a web scraping expert, how can you leverage financial data for potential gain? There are two high-level approaches:

1. For yourself or your company: Build custom web scraping pipelines to gather data from multiple sources. Use it to feed investment models, AI agents, trading algorithms, or analytics pipelines. This is usually highly tailored to your strategies, risk appetite, or operational goals.

2. To sell to financial services: Collect, aggregate, and potentially enrich data from various sources to sell. You can offer broad datasets for many clients or fully customized solutions for a specific customer’s needs.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

How to Approach Financial Data Scraping: Historical vs Real-Time

When it comes to finance, the web is packed with countless data fields and categories (e.g., news, stock prices, filings, analyst reports, and more). It’s a huge industry, and almost anything can be scraped!

At a high level, though, the key distinction for web scraping is simple: the financial data you want to collect is either historical or real-time. That’s what actually makes a difference in the approach to data scraping.

In the following chapters, I’ll dive deeper into each of the two categories of financial data. I’ll cover which fields are most interesting to scrape, where to find them, and how to collect them efficiently and effectively.

For now, start with a brief introduction to historical and real-time financial web data scraping!

Historical Financial Web Data

This includes all past financial data collected from the web, from historical stock prices to inflation rates and archived news. It’s used for analysis supporting long-term investment decisions.

👍 Pros:

• Enables backtesting of investment and trading strategies.

• Easier to scrape, as it isn’t time-sensitive.

• Data itself is stable and doesn’t change over time…

👎 Cons:

• …but the web pages displaying it (e.g., in tables and static charts) can still change, breaking your static parsing logic.

• Misses recent market shifts or breaking events.

• Data completeness varies across websites, often requiring aggregation from multiple sources.

Real-Time Financial Web Data

This includes live financial data extracted from the web, such as stock prices, market news, order books, etc. It’s employed for trading and short-term investment decisions.

👍 Pros:

• Enables fast, data-driven trading decisions.

• Captures live market movements and breaking news.

• Can be passed to AI agents and pipelines directly, as it tends to require minimal preprocessing.

👎 Cons:

• Harder to scrape reliably due to latency constraints and rate limits.

• Requires robust infrastructure for real-time ingestion and analysis, as every second counts.

• Data storage can grow rapidly because new data arrives continuously.

Mastering Historical Financial Data Scraping

As promised, let me guide you through the world of scraping historical financial data from the web.

Main Types of Historical Financial Web Data

The most important types of historical financial data you can retrieve from websites are:

• Historical stock and commodity prices: Open, high, low, close (OHLC) prices and trading volumes for stocks, ETFs, indices, and commodities, used for time-series analysis, modeling, and predictions.

• Summary info and infographics: Stock profiles, key metrics, and past indicators (e.g., P/E, EPS, moving averages), presented in dashboards or visual charts for quick insights.

• Macroeconomic indicators: Inflation, GDP, interest rates, unemployment, CPI, and PCE data, essential for understanding economic cycles and long-term market behavior.

• Financial statements: Company filings (income statements, balance sheets, cash flow), utilized for fundamental analysis and valuation models.

• News data: Archived headlines and press releases analyzed via NLP to correlate past market movements with specific events and sentiment shifts.

• ESG scores and sustainability reports: Historical environmental, social, and governance metrics employed to assess how “green” or ethical a company has been over time.

• Alternative data: Non-traditional datasets like web traffic, social media, satellite imagery (e.g., new headquarters or production plants), or credit card data for early performance signals.

Most Popular Targets

Also, if you’re interested in how to scrape historical data from the Wayback Machine, read my previous guide for this newsletter!

Scraping Techniques

Typical examples of historical financial data include lists of open, high, low, and close prices for a given stock:

Or, another example, the historical returns of a specific index (.e.g, SP500) over time:

These cases fall into the category of table-based data scraping, one of the most common web scraping scenarios. You’re probably already familiar with it, so there’s no need to go too deep here. Scraping older news and media can be slightly more challenging due to the unstructured nature of the target data, but it’s still a simple task.

At a high level, the process for getting historical finance data via web scraping follows a standard workflow:

1. Visit the target web page, either via an HTTP client or a browser automation tool.

2. Parse the page using an HTML parser, either directly or after rendering in a controlled browser.

3. Select the HTML elements of interest and extract the data.

4. Store the scraped data in your desired format (e.g., XLS, CSV, JSON) or in a database.

The main challenges involve generic anti-scraping mechanisms, such as CAPTCHAs, WAFs, IP bans, as well as browser, TLS, and device fingerprinting.

Best Practices

Based on my experience with financial web scraping, especially when focusing on historical data, these are the tips you should apply:

• Normalize and validate data: Standardize formats (dates, currencies, units) and validate across sources to catch inconsistencies early.

• Be cautious with AI parsing: Avoid using AI for automatically parsing structured data (tables, metrics, structured fields). It can introduce subtle errors and hallucinations, so prefer deterministic parsing. Harness AI mainly for retrieving unstructured text like news.

• Store raw HTML snapshots: Always keep the original page HTML. It lets you re-parse data later and extract new signals without re-scraping.

• Avoid single-source bias: When scraping news or market analysis pieces, pull data from multiple sources to reduce bias and improve reliability.

• Handle pagination properly: Many sites split historical data across pages or date ranges. Make sure your scraper fully traverses them all.

• Respect rate limits and retries: Even for historical data, implement retries and throttling to avoid blocks and incomplete datasets.

Check the TWSC YouTube Channel

Understanding Real-Time Financial Data Scraping

This is where things get a bit more interesting. Let me introduce you to real-time financial scraping!

Main Types of Real-Time Financial Web Data

The most relevant types of real-time financial web data are:

• Live price tickers: Continuously updated “last trade” prices and bid/ask spreads for stocks, crypto, and forex, used to detect breakouts and short-term trading opportunities.

• Order book and market depth: Incoming buy/sell orders, liquidity levels, and spreads, fundamental for execution strategies and high-frequency trading.

• Breaking news: Immediate updates and announcements that trigger sentiment models as soon as key figures (CEOs, central banks, governments) release information.

• Corporate event triggers: Monitoring press releases or SEC feeds for earnings surprises, M&A rumors, or sudden executive changes.

• Social media signals: Tracking ticker mentions on platforms like Reddit or X to detect retail-driven momentum, hype cycles, or panic selling in near real time.

• Institutional “whale” activity: Observing large trades or major wallet movements (especially in crypto) to identify where significant capital is flowing.

• Alternative digital signals: Web traffic spikes, app store ranking changes, or “out of stock” alerts on retail sites as proxies for real-world demand.

As you can tell, this category is more varied than historical financial data, including social media tracking and other less conventional practices. Thus, the sources to monitor for live financial web scraping can be less standardized and intuitive.

Most Popular Targets

Scraping Techniques

Imagine applying a traditional scraping pattern to real-time financial data. You send a request to a target site, extract a stock price, and repeat the operation every few seconds or even milliseconds.

The problem is latency. By the time the server responds, the page is rendered or parsed, the target data field is collected, and stored or sent to your pipeline, that piece of data is already outdated.

On top of that, this approach requires a crazy number of requests in a very short time. That increases the risk of triggering rate limiting or even IP bans. You might think proxies solve that through IP rotation, but most proxy networks introduce additional latency, often 2/3/5 seconds per request. In real-time scenarios, that delay is simply not acceptable!

Even if you switch to faster or dedicated proxies, you may end up with a smaller IP pool, which increases the likelihood of those IPs getting blocked.

A more advanced idea is to rely on browser automation and keep a page open, capturing updates as they happen. This is smarter, but still problematic. Long-lived sessions with little or no user interaction are highly suspicious and can easily trigger anti-bot systems. Plus, browser automation at scale tends to be flaky, not really reliable for persistent connections.

Long story short, scraping real-time financial data this way quickly turns into a losing game.

The solution? Stop targeting the data presentation layer in HTML and instead go directly to the data source!

API/WebSocket Scraping as The Solution

Web pages showing real-time financial data aren’t doing anything magical. Behind the scenes, they either poll APIs at regular intervals or (more commonly) maintain a persistent connection via WebSockets to receive continuous updates. The page simply renders that incoming data.

As a result, a much better approach is to intercept and replicate those data flows. You can do this through AJAX/API request inspection or WebSocket sniffing. Open the browser developer tools, go to the “Network” tab, and check where the data is coming from.

If it’s an API call, you’ll see it under the “Fetch/XHR” tab:

If it’s a WebSocket, you’ll find it under the “Socket” tab:

Once identified, replicate those API calls or connect directly to the WebSocket in your scraping script. This gives you access to near real-time financial data in a structured format (typically JSON) without the overhead of parsing HTML.

Of course, that’s not trivial. WebSockets require proper anti-bot bypass, and APIs may still enforce rate limits, tracking, and TLS fingerprinting protections. However, this approach is generally faster, more reliable, and much easier to maintain than scraping rendered pages!

And What About Live News or Social Media Scraping?

When it comes to news, if available, it makes sense to connect to public RSS feeds exposed by websites to monitor updates. This allows you to trigger scraping only when new and relevant content is published, instead of constantly polling pages unnecessarily.

Otherwise, you can build a polling mechanism that periodically checks news sites, social media platforms, and similar sources to capture fresh data. In these cases, you usually can’t rely on techniques like API or WebSocket scraping, as that’s not how those platforms fetch data.

Instead, you need a solid and robust infrastructure built around speed and efficiency: fast connections, high-quality proxies, optimized parsing, and lightweight requests. The goal is to minimize latency while maintaining reliability at scale.

Best Practices

Scraping real-time financial data is a demanding art, but it becomes easier with the following best practices:

• Prefer APIs and WebSockets over HTML parsing: Whenever possible, save time by extracting data directly from the underlying APIs or WebSocket streams utilized by web pages instead of scraping data from rendered pages.

• Choose clean, structured sources: Prioritize endpoints that return well-formatted JSON to minimize preprocessing and reduce latency.

• Stream data into pipelines immediately: Send incoming data directly to processing pipelines for real-time insights, while storing it in parallel for later analysis.

• Use specialized AI for sentiment analysis: Prefer AI/ML models tuned for finance/social media, as Reddit and X content often include slang, memes, and non-standard language.

• Optimize browser automation: Configure Playwright, Selenium, or similar browser automation tools to block images, stylesheets, and fonts. This reduces bandwidth usage and significantly speeds up rendering time.

• Design for low latency: Optimize your stack (async requests, streaming ingestion, fast JSON parsers) to minimize delays, as even milliseconds matter.

• Prefer high-quality premium proxies: Count on proxy providers with a proven track record of fast, stable connections to minimize latency and avoid disruptions.

• Time-synchronize everything: Append timestamps to all scraped data to enable time-series analysis and accurately reconstruct events.

• Build fault-tolerant systems: Expect disconnections (especially with WebSockets) and issues, so add reconnection logic and configure fallback data sources.

Need help with your scraping project?

Top 5 Open-Source Financial Web Scraping Libraries

Below is a selected set of interesting, fully open-source libraries, packages, and projects for simplified financial web scraping:

Conclusion

Here, I’ve gone through the rabbit hole of financial web scraping, the task of collecting finance-related data from the Internet. This is one of the main use cases of corporate web scraping, powering enterprise data pipelines for decision-making and market analysis.

As you’ve seen, the main difference in the approach comes down to whether you’re targeting historical or real-time data. The first follows standard web scraping practices you’re likely already familiar with. The second is trickier and requires more advanced techniques.

I hope you found this helpful and insightful. If you have questions, feel free to share them in the comments below!

This time, we wanted AWS WAF on its own, in front of a target with nothing else in front of it, and we wanted to see what changes when the 2024 Scrapy-Playwright stack is replaced with the 2026 toolbox.

The target we picked is imdb.com. It is an Amazon subsidiary, runs a standard AWS WAF deployment, and Wappalyzer confirms that there are not others antibot on the website. That makes IMDB a perfect use case for our article.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Today we’ll test three Python HTTP clients with strong TLS fingerprint impersonation: curl_cffi, the newer httpx-curl-cffi, and Rust-backed rnet. Each one produces a TLS handshake indistinguishable from real Chrome. Is that enough to scrape an AWS WAF target without spinning up a browser? And if not, what is the smallest browser step that gets us past the gate so the rest of the work can run on a cheap HTTP client?

The tools we used

Four libraries are in scope. Three are HTTP-only, one runs a real browser.

curl_cffi is a Python binding for the curl-impersonate patched curl. It exposes a requests-like API and ships impersonation profiles for recent Chrome, Firefox, and Safari builds and works at the TLS layer. JA3 and JA4 fingerprints match the impersonated browser, along with HTTP/2 settings and header order. We tested with chrome142, the latest Chrome profile in version 0.14.0.

httpx-curl-cffi is a transport for httpx that delegates the actual HTTP work to curl_cffi. While it does not add new fingerprinting capability, it implements the httpx programming model: sync Client, async AsyncClient, event hooks, the same response object you get from the rest of an httpx-based codebase. We tested with the Chrome profile and default_headers=True.

rnet is a Rust HTTP client with Python bindings. It implements its own impersonation stack rather than wrapping curl-impersonate. The enum rnet.Impersonate exposes a wide range of Chrome, Firefox, Safari, Edge, Opera, and OkHttp profiles. We tested with Chrome137.

Scrapling is the only browser-driven tool in the set. Our Scrapling: A Complete Hands-On Guide goes through the library in depth, with Cloudflare as the test target. Its StealthyFetcher drives a stealth-patched Chromium that runs JavaScript and applies fingerprint countermeasures. The library README only advertises Cloudflare Turnstile, but the same machinery handles AWS WAF’s challenge too.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

How AWS WAF protects IMDB

A quick intro of the system helps interpret the results that follow. AWS WAF is not a dedicated anti-bot platform like DataDome or Kasada. It is a general-purpose web application firewall with a bot-control module that operators can enable per rule. When the bot-control rule is in challenge mode, AWS WAF inserts a single JavaScript gate at the start of a session.

A request without a valid cookie returns HTTP 202 with x-amzn-waf-action: challenge and a short HTML body. The body contains window.gokuProps containing three base64 blobs (key, iv, context), a <script src> pointing to a customer-specific URL on *.token.awswaf.com, and a small inline script that calls AwsWafIntegration.saveReferrer(), AwsWafIntegration.checkForceRefresh(), and AwsWafIntegration.getToken(). The remote challenge.js tests the browser environment, posts a validation payload back to AWS, and on success, the response sets Set-Cookie: aws-waf-token=.... The inline script then reloads the page, and the second request, now carrying the token, gets the real content.

This works very differently from systems that score every request. Once the token is in our jar, AWS WAF lets us through with no further behavioral checks beyond IP reputation and rate limits.
What we want to discover with this article is if we’re able to bypass AWS WAF with “convincing” requests, with a proper TLS fingerprint and set of headers, or if we need a JS rendering engine.

Check the TWSC YouTube Channel

Test setup

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 104.IMDB. If you’re not able to access the repository, please use this form to request access.

The libraries we pinned at the time of writing are curl_cffi==0.14.0, httpx==0.28.1, httpx-curl-cffi==0.1.5, rnet==2.4.2, scrapling==0.4.7. Python is 3.11.

Each HTTP test creates a GET against two URLs: the IMDB home page

https://www.imdb.com/

and a title page https://www.imdb.com/title/tt0111161/. We use two URLs to confirm the challenge fires the same way on both, not only on one entry point. We do not follow redirects (follow_redirects=False) because the AWS WAF response is a 202 with content rather than a redirect, and we want to see it raw.

We capture status code, HTTP version, the full response headers, any cookies, body length, and the first 600 characters of the body, and we saved everything to JSON under aws_waf_imdb/responses/ for later inspection.

The baseline probe in probe_plain.py uses an unmodified httpx.Client(http2=True) with a generic Chrome User-Agent header and the standard Accept headers. This is the control: no TLS impersonation, no fingerprint trickery, just a normal Python HTTP client.

Read more

The web contains an extraordinary volume of information, the majority of which is in textual form. Blogs, forums, and newsletters alone generate millions of words of domain-specific knowledge every week. And they’re not the only sources of text on the web.

When you want to get insights from that kind of data, successfully extracting it from the web is only half of the battle, even now that LLMs can use vision to scrape complex visual layouts. The second part of the challenge is structuring this data to get it ready for analytics. Why? Because when you point a scraper at a news article, you get back a wall of text. But you cannot query it. You cannot aggregate it. You cannot feed it reliably into a machine learning pipeline or a database without significant preprocessing.

This article addresses the preprocessing problem of unstructured text when you scrape it from the web. It traces the evolution of solutions from classical NLP to large language models, identifies where each approach breaks down, and proposes a practical architectural solution.

Let’s get into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What “Unstructured” Really Means in Practice

Unstructured text refers to content that carries no machine-readable schema. The information exists in the data you retrieved from the web, but no field boundaries exist, no consistent labels, and no guaranteed position for any given fact.

The following schema represents the difference between unstructured and structured text (machine-readable schema):

Let’s consider three concrete scraping targets to illustrate what this costs you in practice.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

News Articles: When Signals Are Buried in Noise

Consider you scraped a Reuters article about an ECB rate decision. The text you get back from the scraper could be something as follows:

European Central Bank decides on rates.
Listen to this article. 2 min audio. 
You might also like: Eurozone inflation hits 3-year low. 
Christine Lagarde announced Thursday a 25 basis point reduction, bringing the main 
refinancing rate to 3.40%. 
SPONSORED: Track macro events with Bloomberg Terminal. 
The decision was widely anticipated after last month's CPI print. Share this article. 
4 comments. John M. writes: this was priced in already

Your raw text contains the article body, a teaser for a related story, a sponsored insertion, and reader comments. The fact you want is buried in there: the ECB cut its main refinancing rate to 3.40% on a specific date. But your extractor gets the full content.

Such a wall of text, which, generally speaking, is way bigger than this and is useless for analytics purposes without preprocessing.

Financial Newsletters: When “Just Under Two Percent” Breaks Your Aggregation

Suppose you scrape a financial newsletter to extract an updated macroeconomic forecast. You need to capture a specific fact. Something like “Goldman Sachs revised its 2026 US GDP growth forecast down to 1.8%”. Your scraper captures the entire page output, which is similar to an article. Similarly to the previous example, the resulting raw text mixes the core facts with boilerplate and unrelated news:

Market Daily Newsletter. November 12.
Jan Hatzius (Goldman Sachs) and his team were out with a note early Tuesday.
SPONSORED: Get 50% off your trading fees today. 
They see tariffs shaving roughly 0.7 points off the baseline. Meanwhile, 
European markets rallied on ECB news. 
Read our full coverage of the Eurozone here.
The revised number now sits just under two percent for the full year.
Subscribe for premium insights.

The text distributes the target fact across the entire document. Also, the wording “just under two percent” requires numerical understanding to say that the text refers to the actual number you were searching for, that is, an exact 1.8%.

Now, imagine generalizing this after scraping hundreds of financial news and newsletters to regroup the information to summarize the numbers. Getting insight would be impossible. Why? Because some sources will give you the actual information you want (growth forecast down to 1.8%), others will use different phrasing to define the trend (”An expected growth under 2 percent”, “a slightly shrinking trend”, etc).

Without a way to create a structure for such data, you can’t get any insights from it.

Job Posting Offers: They Are Always Messier Than They Look

Consider the case when you want to scrape job offers to get an idea of what the market is paying on average for a specific position, given the expected technical skills, and considering the same day-to-day activity. Job offers can have the following ambiguities:

• A sentence might read “3+ years of experience with Python”. This establishes a floor and ignores a ceiling. Alternatively, the text might read “Senior-level candidates only”. This uses qualitative seniority as a proxy for an exact quantitative number.

• Salary breaks in a different direction. One posting can say “$120,000 - $145,000 base”. Another can be “competitive compensation commensurate with experience”. A third could be“€100,000”, which you need to convert to dollars to make an actual comparison.

• Employment type can introduce further ambiguity and difficulties. “Full-time”, “FTE”, “permanent”, and “direct hire” basically mean the same thing but are written differently. Also, the text might specify the role is “Hybrid”, which means multiple different things across companies. It could mean two days in the office. It could mean occasional travel with headquarters-optional rules.

When sites get tough, skip the heavy lifting. Get clean, structured CSV datasets, ready for Excel, BI or your apps

Find your dataset

How Classical NLP Tried to Solve This (and Where It Stopped)

Before large language models were released, the standard answer to this problem was Natural Language Processing. The classical NLP toolkit gave developers a set of tools that could, with enough effort, extract meaningful structure from text using different, but often interconnected, processes like the following:

• Named Entity Recognition (NER): NER is a process used in NLP to extract entities from text corpora. It can particularly identify spans of text as persons, organizations, locations, or dates. An NLP model trained on news corpora, for example, is able to scan an article and tag “Jane Doe” as a person and “Washington D.C.” as a geopolitical entity.

• Part-of-speech tagging: Is a process in which NLP models can identify nouns, verbs, and adjectives. This enables the downstream logic to focus on the right parts of a sentence.

• Dependency parsing: Maps grammatical relationships between words, helping to extract which subject performed which action on which object.

• Relation extraction: Identifies when two co-occurring entities have a specific relationship. For example, a person who was affiliated with an organization, or an event that occurred in a specific location.

Libraries like spaCy, Stanford NLP, and NLTK made these processes largely accessible. But they work well for well-defined, narrow tasks on consistent text domains. The problems and limitations of this solution appear quickly at the edges:

• Domain shift breaks everything: A NER model trained on news articles performs poorly on scientific abstracts. A model tuned for English financial text fails on multilingual content. In other words, every new domain requires retraining, re-labeling, and re-evaluation. These processes are very costly, both in terms of money and time.

• Context is invisible: Classical NLP models operate at the token and sentence level. They have no mechanism for understanding that “Apple” in a technology article refers to a corporation, while “apple” in a nutrition blog refers to a fruit. Disambiguation requires hand-crafted rules or separate classification layers bolted on top (which, again, is costly).

Before NLP, you could basically only use regex (with all the difficulties associated with manually cleaning data, standardizing it, and…using regex!). So, NLP was a genuine (big) step forward: it made large-scale text analysis possible in ways that pure pattern matching never could (which is a way to find patterns in scraped data using AI). But it still required substantial domain expertise, constant maintenance, and produced results that were narrow, fragile, and difficult to generalize.

Check the TWSC YouTube Channel

The Modern Solution: LLMs as Universal Structure Extractors

Large language models fundamentally changed the extraction problem. On the side of the underlying technology, a classical NLP model learns the statistical patterns inside the text. An LLM, instead, learns to understand language. This distinction matters enormously because it opened the doors to the following:

• Context disambiguation that works out of the box: Feed an LLM with a paragraph from a technology article containing the word “Apple” and it will correctly identify it as a company. Feed it with a paragraph from a recipe blog, and it will correctly identify it as a fruit. No separate disambiguation layer. The model resolves ambiguity the same way a human reader does: by reading the surrounding context.

• Semantic equivalence that is understood, not computed: An LLM knows that “$40,” “forty dollars,” “40 USD,” and “forty bucks” all express the same value. You don’t need to instruct it to understand that.

• Implicit information that becomes accessible: A sentence like “the study, conducted over three months at a Boston hospital, found no significant effect” contains a location, a duration, and a finding. An LLM can extract all three without requiring the text to follow any particular structure.

• Domain generalization that requires no retraining: The same LLM that extracts entities from political news articles can extract findings from scientific abstracts, event mentions from cultural journalism, and source attributions from investigative reporting. You just need to change the prompt, not the model.

The practical workflow becomes straightforward:

• You scrape unstructured text from the web.

• You pass the content to an LLM with a prompt that describes what you want to extract.

• The model returns a response.

• You use that response downstream.

This process works. But using LLMs alone introduces a different class of problems:

• Output format is not guaranteed: Ask an LLM to return a price, and it might return $40 in one run, 40 dollars in another, and 40 USD in a third. The model understands the value when it retrieves it from scraped content. But it does not guarantee how it expresses that value unless you explicitly constrain it.

• Required fields can go missing: If the article you extracted the content from does not mention a publication date, the model might omit the field, return null, or return "not mentioned", or invent a plausible date (which is way worse). Each behavior is different, and none of them is predictable without enforcement.

• Hallucination is a real risk: When the model is uncertain, it always generates a plausible answer. For extraction tasks, that means it can invent entity names, fabricate statistics, or fill in missing information with confident-sounding fiction. Without validation, these errors pass into your data, creating issues at the analytics level.

Generalizing all of this, you also get scalability issues because you have no consistency guaranteed. A pipeline processing 10,000 articles requires every output to follow the same schema. But a model that returns slightly different structures across runs cannot feed a database reliably without significant error handling.

In other words, LLMs provide you with the understanding that NLP lacked. But they do not, on their own, provide the structural guarantees that production pipelines require.

Need help with your scraping project?

How to Get Semantic Power and Structural Guarantees at the Same Time: A Practical Approach

One possible solution to the unpredictability of LLM outputs is to separate the two concerns that these models conflate: semantic understanding and structure enforcement.

To do so, you can:

• Use the LLM for what it does well: reading text, resolving ambiguity, extracting meaning, and normalizing inconsistent expressions.

• Use specific libraries to define schemas, enforce types, validate outputs, and reject malformed data before it enters your pipeline.

Below is how this solution works, at a high level:

Let’s see how to implement this process and how the two approaches differ in practice.

The Baseline Approach: A Direct LLM Call (and What It Gives You)

Consider the following content that can come from scraping a news article:

Fed Signals Caution as Inflation Data Disappoints

By Sarah M. Connelly | April 14, 2026 | Economics & Policy

The Federal Reserve signaled on Monday that it remains in no rush to cut interest rates,
after fresh inflation data showed consumer prices rose more than expected last month.
Jerome Powell, speaking at a conference in Washington, said the central bank needs
"greater confidence" that inflation is moving sustainably toward its two-percent target
before reducing borrowing costs.

The Consumer Price Index climbed 3.5 percent in March, up from 3.2 percent in February,
according to the Labor Department. Economists polled by Reuters had forecast a reading
of 3.3 percent. Core CPI, which strips out food and energy, rose 3.8 percent year-over-year.

Markets reacted sharply. The S&P 500 fell nearly one point five percent by midday,
while the yield on the 10-year Treasury note jumped to four point six percent.
Goldman Sachs revised its forecast, pushing back its expected first rate cut from June
to September. JPMorgan analysts said two cuts in 2026 now look "optimistic."

Powell emphasized that the Fed is not considering rate hikes at this stage, but stressed
that the path back to two percent inflation "may take longer than previously thought."
The next Fed meeting is scheduled for the first week of May

To directly pass it to a GPT model, asking it for a precise output, you can use the following code:

import os
import json
from openai import OpenAI

# Scraped content
SCRAPED_TEXT = """
Fed Signals Caution as Inflation Data Disappoints

By Sarah M. Connelly | April 14, 2026 | Economics & Policy

The Federal Reserve signaled on Monday that it remains in no rush to cut interest rates,
after fresh inflation data showed consumer prices rose more than expected last month.
Jerome Powell, speaking at a conference in Washington, said the central bank needs
"greater confidence" that inflation is moving sustainably toward its two-percent target
before reducing borrowing costs.

The Consumer Price Index climbed 3.5 percent in March, up from 3.2 percent in February,
according to the Labor Department. Economists polled by Reuters had forecast a reading
of 3.3 percent. Core CPI, which strips out food and energy, rose 3.8 percent year-over-year.

Markets reacted sharply. The S&P 500 fell nearly one point five percent by midday,
while the yield on the 10-year Treasury note jumped to four point six percent.
Goldman Sachs revised its forecast, pushing back its expected first rate cut from June
to September. JPMorgan analysts said two cuts in 2026 now look "optimistic."

Powell emphasized that the Fed is not considering rate hikes at this stage, but stressed
that the path back to two percent inflation "may take longer than previously thought."
The next Fed meeting is scheduled for the first week of May.
"""

# Define LLM client
raw_client = OpenAI(api_key=os.environ.get("YOUR_OPENAI_API_KEY"))

# Define prompt for the LLM
raw_prompt = """
Extract the following information from the article below and return it as JSON:
- title
- author
- publication_date
- mentioned_organizations
- cpi_march_value
- key_claim
- market_sentiment

Article:
""" + SCRAPED_TEXT

# Get response from LLM
raw_response = raw_client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": raw_prompt}]
)

raw_output = raw_response.choices[0].message.content

# Print results
print(raw_output)

The result will be as follows:

{
  "title": "Fed Signals Caution as Inflation Data Disappoints",
  "author": "Sarah M. Connelly",
  "publication_date": "April 14, 2026",
  "mentioned_organizations": [
    "Federal Reserve",
    "Labor Department",
    "Reuters",
    "Goldman Sachs",
    "JPMorgan"
  ],
  "cpi_march_value": "3.5 percent",
  "key_claim": "The Federal Reserve is in no rush to cut interest rates and needs greater confidence that inflation is moving sustainably toward its two-percent target before reducing borrowing costs.",
  "market_sentiment": "Negative"
}

Now, at first sight, this seems good. The prompt asked the GPT model to create a JSON file with specific values, and the model was able to do so. But two major problems affect the next steps when analyzing this data. They are:

• The publication date is reported as “April 14, 2026”. This is not represented in ISO 8601 format and will break any date parser.

• The CPI is reported as “3.5 percent”, which is a string. Not a number or a float, which is what is required for such data if you want to further analyze it (without any intermediate steps).

So, the LLM was able to give structure to an unstructured text, after being specifically prompted to do so. But it failed at providing the data in the right format. To do so, you have to provide specific guidance to the model.

What Changes When You Define The Schema

To have guarantees on the output format, you can use the following code:

import os
import json
import instructor
from openai import OpenAI
from pydantic import BaseModel, Field
from typing import Optional, Literal

# Scraped content
SCRAPED_TEXT = """
Fed Signals Caution as Inflation Data Disappoints

By Sarah M. Connelly | April 14, 2026 | Economics & Policy

The Federal Reserve signaled on Monday that it remains in no rush to cut interest rates,
after fresh inflation data showed consumer prices rose more than expected last month.
Jerome Powell, speaking at a conference in Washington, said the central bank needs
"greater confidence" that inflation is moving sustainably toward its two-percent target
before reducing borrowing costs.

The Consumer Price Index climbed 3.5 percent in March, up from 3.2 percent in February,
according to the Labor Department. Economists polled by Reuters had forecast a reading
of 3.3 percent. Core CPI, which strips out food and energy, rose 3.8 percent year-over-year.

Markets reacted sharply. The S&P 500 fell nearly one point five percent by midday,
while the yield on the 10-year Treasury note jumped to four point six percent.
Goldman Sachs revised its forecast, pushing back its expected first rate cut from June
to September. JPMorgan analysts said two cuts in 2026 now look "optimistic."

Powell emphasized that the Fed is not considering rate hikes at this stage, but stressed
that the path back to two percent inflation "may take longer than previously thought."
The next Fed meeting is scheduled for the first week of May.
"""

# Validation schema
class ArticleExtraction(BaseModel):
    title: str = Field(description="The article's headline")
    author: Optional[str] = Field(description="Full name of the author if explicitly mentioned")
    publication_date: Optional[str] = Field(description="Publication date in ISO 8601 format (YYYY-MM-DD)")
    mentioned_organizations: list[str] = Field(description="All organizations referenced in the article")
    cpi_march_value: Optional[float] = Field(description="CPI value as a float (e.g. 3.5)")
    key_claim: str = Field(description="The central argument or finding of the article in one sentence")
    market_sentiment: Literal["positive", "negative", "neutral"] = Field(
        description="Overall market sentiment expressed in the article"
    )

structured_client = instructor.from_openai(OpenAI(api_key=os.environ.get("YOUR_OPENAI_API_KEY")))

extraction = structured_client.chat.completions.create(
    model="gpt-4o",
    response_model=ArticleExtraction,
    messages=[
        {
            "role": "user",
            "content": f"Extract structured information from the following article:\\n\\n{SCRAPED_TEXT}"
        }
    ]
)

print(extraction.model_dump_json(indent=2))

print("\\n" + "=" * 60)
print("INSPECTION OUTPUT")
print("=" * 60)
for field, value in extraction.model_dump().items():
    print(f"  {field}: {repr(value)}  →  type: {type(value).__name__}")

The above code leverages two fundamental libraries:

• Pydantic: This is a Python data validation library. You define a schema as a Python class, declare the fields and their types, and Pydantic enforces that any data you put into that class matches what you declared.

• Instructor: This is the bridge between Pydantic and the LLM. The core problem it solves is that LLMs’ APIs return text, but Pydantic validates Python objects. So, something has to sit in the middle, take the LLM’s response, parse it into the structure your Pydantic model expects, and retry the call if the output doesn’t validate. That’s what Instructor does. Without Instructor, you would have to manually prompt the model to return JSON, parse that JSON yourself, handle malformed responses, write retry logic, and coerce types by hand.

By using these two libraries, the ArticleExtraction() class does the following:

• Type enforcement: Defines cpi_march_value as a float. This guarantees the model will return an actual number) instead of a string (3.5 instead of "3.5 percent" as the previous example).

• Controls formatting and vocabulary: The Literal type on market_sentiment restricts the LLM’s output to "positive", "negative", or "neutral". The model cannot invent new categories. Similarly, the description for publication_date explicitly demands the ISO 8601 format.

• Built-in prompting: The Field(description="...") parameters serve a dual purpose. First, they document the code for developers. Secondly, under the hood, the Instructor library feeds these exact descriptions to the LLM as targeted instructions. This ensures the model understands exactly what “key claim” or “publication date” means in this context.

• Graceful omissions: Wrapping fields like author in Optional[...] gives the model permission to safely return a null value if the information isn’t present in the scraped text. This highly reduces the risk of hallucinations.

The JSON output is as follows:

{
  "title": "Fed Signals Caution as Inflation Data Disappoints",
  "author": "Sarah M. Connelly",
  "publication_date": "2026-04-14",
  "mentioned_organizations": [
    "Federal Reserve",
    "Labor Department",
    "Reuters",
    "Goldman Sachs",
    "JPMorgan"
  ],
  "cpi_march_value": 3.5,
  "key_claim": "The Federal Reserve remains cautious about cutting interest rates because inflation has not yet shown sufficient progress toward its two-percent target.",
  "market_sentiment": "negative"
}

As you can see, now the CPI is a float, and the publication date is in ISO 8601.

The inspection output is the following:

============================================================
INSPECTION OUTPUT
============================================================
  title: 'Fed Signals Caution as Inflation Data Disappoints'  →  type: str
  author: 'Sarah M. Connelly'  →  type: str
  publication_date: '2026-04-14'  →  type: str
  mentioned_organizations: ['Federal Reserve', 'Labor Department', 'Reuters', 'Goldman Sachs', 'JPMorgan']  →  type: list
  cpi_march_value: 3.5  →  type: float
  key_claim: 'The Federal Reserve remains cautious about cutting interest rates because inflation has not yet shown sufficient progress toward its two-percent target.'  →  type: str
  market_sentiment: 'negative'  →  type: str

This validation helps immediately see that the data types are correct.

Conclusion

In this article, you learned what unstructured text actually costs a data pipeline. You saw how classical NLP made structured extraction possible but fragile, and how LLMs removed the domain constraints that NLP never solved. You also learned why LLMs alone are not enough and saw a practical solution to provide “guardrails” for LLMs so that their output follows a defined schema.

So, let us know: how are you managing unstructured text after you scraped it?

In this article, you’ll find out this and more. I’ll walk you through a complete guided example of how to use it, and break down its (Spoiler: undoubtedly serious) limitations.

An Introduction to the Cloudflare Crawl Endpoint

Before exploring the technical aspects behind the Cloudflare /crawl endpoint and seeing it in action, let me first give you some context!

What Is the Cloudflare /crawl Endpoint?

The /crawl endpoint is a new addition to Cloudflare’s REST APIs. Its goal is to crawl an entire website (or just a portion of it) starting from a single URL.

Note: The Crawl endpoint is currently in beta and was introduced on March 10, 2026, as highlighted in the Cloudflare changelog.

Under the hood, it automatically discovers and visits new pages, rendering them in a headless browser. It returns the discovered content as HTML, Markdown, or structured JSON, making it ideal for RAG pipelines, monitoring, or dataset creation.

As I’ll dive into later, it respects robots.txt and doesn’t bypass bot protection or captchas. Thus, it’s designed as a compliant approach to web crawling!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

How It Works at a High Level

At a high level, the /crawl endpoint involves a two-step flow:

1. You kick off an asynchronous crawl job, passing a starting URL. Cloudflare returns a job ID.

2. You use that job ID to periodically check the job’s status or fetch results as they become available, following typical polling behavior.

Important: A crawl job can run for up to seven days! Results remain available for 14 days after completion, after which the job data is deleted.

Behind the scenes, the crawler expands outward from the starting URL. By default, the API follows a clear order:

1. The initial page.

2. Sitemap URLs.

3. Links discovered within pages.

Still, you can tweak that depending on whether you want to prioritize sitemaps, page links, or both.

Supported Use Cases

The officially promoted use cases for the Cloudflare /crawl API are just two:

• Creating knowledge bases or training AI systems (like RAG applications) using up-to-date web content.

• Collecting and analyzing content across multiple pages for research, summarization, or monitoring purposes.

Pricing

Compared to most other web crawling or discovery APIs on the market, Cloudflare’s /crawl API doesn’t charge by the number of pages. Instead, costs are based on resource usage, which depends on whether you enable the headless browser rendering feature.

If headless rendering is active, pricing follows the Browser Rendering model:

If rendering isn’t active, pricing follows the Workers model:

Yeah, I know… It’s honestly a bit confusing, and it’s almost impossible to predict the exact cost of a crawling task. The good news? You can test it for free!

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Cloudflare Crawl Endpoints: Technical Analysis

Now that you know what Cloudflare is and what it brings to the table, it’s time to better understand its functioning, strengths, and limitations.

Endpoint Presentation

The Cloudflare Crawl API is built around two main endpoints. Both share the same base URL:

https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl

Where <ACCOUNT_ID> is your Cloudflare account ID.

1. Initiate the Crawl Job (POST)

To start a new crawl, you need to send a POST request with the target URL (and optional parameters like depth, rendering mode, etc.) as below:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl' \
  -H 'Authorization: Bearer <CLOUDFLARE_API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{ "url": "https://example.com" }'

Keep in mind that the endpoint supports several parameters, allowing you to greatly customize the crawling behavior, output format (JSON, HTML, or Markdown), rendering options, caching, and more. Check out the full list of supported body parameters for all available options.

Cloudflare immediately returns a job ID that you’ll use to track or retrieve results. A possible response looks like this:

{
  "success": true,
  "result": "9f1c2d3a-4b5e-6f7a-8c9d-0e1f2a3b4c5d"
}

The UUID in the result field is the Crawl job ID you’ll use to poll for updates.

2. Request Crawl Results (GET)

Once the crawl is running, make a GET request with the job ID to check the status or fetch results:

curl -X GET 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl/<JOB_ID>' \
  -H 'Authorization: Bearer <CLOUDFLARE_API_TOKEN>'

Here, the <JOB_ID> placeholder is the UUID retrieved before from the result field.

The response either includes a status field like this:

{
  "result": {
    "id": "3e7a1c92-b4d8-4f6e-9a21-6c0d5b8e2f14",
    "status": "running"
    // ...
  }
}

The possible status values are: running, completed, errored, or one of several cancellation states (cancelled_due_to_timeout, cancelled_due_to_limits, cancelled_by_user).

Or, once the job is completed, calling the API returns the full results in the records field:

{
  "result": {
    "id": "3e7a1c92-b4d8-4f6e-9a21-6c0d5b8e2f14",
    "status": "completed",
    "browserSecondsUsed": 98.3,
    "total": 12,
    "finished": 12,
    "records": [
      {
        "url": "https://example.com/",
        "status": "completed",
        "markdown": "# Example Domain\nThis domain is for use in illustrative examples...",
        "metadata": {
          "status": 200,
          "title": "Example Domain",
          "url": "https://example.com/"
        }
      },
      {
        "url": "https://example.com/about",
        "status": "completed",
        "markdown": "## About\nLearn more about this example site...",
        "metadata": {
          "status": 200,
          "title": "About - Example Domain",
          "url": "https://example.com/about"
        }
      }
      // additional entries omitted for brevity...
    ],
    "cursor": 10
  },
  "success": true
}

Note that the response will vary based on the specified query parameters. For example, you can filter by specific statuses, limit the number of results, and navigate through them using a pagination system.

Features

Below is a list of the main, most relevant capabilities provided by the Cloudflare Crawl API:

• Asynchronous crawl jobs: Trigger crawling jobs and poll results when they are ready, enabling non-blocking, large-scale crawling workflows.

• Automatic URL discovery: Finds pages from the starting URL, sitemaps, and in-page links, with configurable source control.

• Flexible output formats: Returns HTML, Markdown, or structured JSON. JSON leverages Workers AI for schema-driven data extraction.

• Headless browser rendering: Control JavaScript execution with render: true or perform fast static HTML fetches with render: false.

• Fine-grained crawl control: Configure limit, depth, and URL inclusion/exclusion with the includePatterns/excludePatterns fields.

• Incremental and cache-aware crawling: Use modifiedSince and maxAge parameters to avoid re-fetching unchanged content, optimizing performance and cost.

• Advanced filtering and pagination: Retrieve results using limit, cursor, and status filters to handle large datasets efficiently.

• Authentication and custom headers: Supports HTTP auth, cookies, and custom headers for crawling protected or API-driven content.

• Dynamic content handling: Wait for JS-rendered content using gotoOptions and waitForSelector, ideal for SPAs and interactive pages.

• Resource skipping for performance: Optionally block images, media, fonts, or stylesheets to speed up crawling and reduce unnecessary bandwidth usage.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

Limitations

Cloudflare’s /crawl API also comes with several important limitations, such as:

• Respects bot protection: The crawler can’t bypass CAPTCHAs (including Turnstile challenges) or Cloudflare bot protections. As a rule of thumb, sites protected via Cloudflare Bot Management or other WAFs tend to block crawling tasks entirely, limiting automated access and leading to incomplete datasets.

• Fixed User-Agent: The /crawl endpoint sets a non-customizable User-Agent value (CloudflareBrowserRenderingCrawler/1.0). You can’t change it, which may cause sites to block requests or serve different content based on the User-Agent.

• Content Signals enforcement: If a site disallows AI usage via Cloudflare Content Signals, crawl requests for those purposes are rejected with a 400 Bad Request error. Even if the site allows other uses, attempts to crawl disallowed categories will fail, limiting AI-specific data collection.

• Rate limiting and crawl pacing: Sites with strict rate limits can slow down crawling. The crawler respects the robots.txt Crawl-delay directive and implements backoff. Large crawls may need to be split into smaller jobs to avoid throttling or skipped URLs.

• Browser usage limits and job cancellation: Accounts on Workers free plans are capped at 10 minutes of browser time per day. Exceeding this limit results in a cancelled_due_to_limits status. To avoid that, you can upgrade to a paid plan.

How to Use the Cloudflare Crawl Endpoint: Step-by-Step Tutorial

In this guided section, I’ll show you how to use the Cloudflare Crawl Endpoint to crawl a website in Python. The target site will be the “Quotes to Scrape” sandbox. The goal here is to demonstrate how to use the API, rather than actually collecting relevant data.

Follow the instructions below!

Prerequisites

To follow this tutorial section, make sure you have:

• Your Cloudflare account ID at hand.

• A Cloudflare API token with the “Browser Rendering - Edit” permission.

For the sake of simplicity and to keep this tutorial concise, I’ll assume you already have a Python project set up with requests installed. That said, you can use any programming language and any HTTP client, because the high-level logic remains the same.

Step #1: Set Up the Configurations

Start by importing the required libraries and reading the necessary secrets (your Cloudflare API token and account ID). Use these secrets to prepare the Cloudflare Crawl base URL and headers. Also, specify the starting target URL as a constant.

import requests
import time
import json

# The Cloudflare secrets required for authenticating Crawl API calls
CLOUDFLARE_API_TOKEN = "<YOUR_CLOUDFLARE_API_TOKEN>"
CLOUDFLARE_ACCOUNT_ID = "<YOUR_CLOUDFLARE_ACCOUNT_ID>"

# The base URL used for all Crawl API calls
CLOUDFLARE_CRAWL_BASE_URL = f"https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/browser-rendering/crawl"

# Common headers shared by all API calls
HEADERS = {
    "Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}",
    "Content-Type": "application/json"
}

# The URL to crawl
START_URL = "https://www.ssense.com/en-us/men/product/acne-studios/silver-folded-leather-wallet/18169981"

Tip: In a production script, read the Cloudflare API token and account ID from environment variables rather than hardcoding them.

Step #2: Trigger the Crawling Job

Define a start_crawl() function to send a POST request to Cloudflare’s Crawl API:

def start_crawl(start_url):
    # Customize according to your needs
    payload = {
        "url": start_url,
        "limit": 20,
        "depth": 2,
        "formats": ["markdown"],
        "render": True
    }

    response = requests.post(CLOUDFLARE_CRAWL_BASE_URL, headers=HEADERS, json=payload)
    response.raise_for_status()

    job_id = response.json()["result"]
    return job_id

This creates a new crawling job for the target URL. Then, it returns a job_id that identifies this specific crawl.

Tip: In a production-level script, make the payload object configurable via function input arguments for greater flexibility and reusability.

Step #3: Poll Over the Job

Next, add a wait_for_completion() function to repeatedly check the job status every few seconds until the crawl finishes or times out:

def wait_for_completion(job_id, max_attempts=60, delay=5):
    status_url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}?limit=1"

    for attempt in range(max_attempts):
        res = requests.get(status_url, headers=HEADERS)
        res.raise_for_status()

        status = res.json()["result"]["status"]
        print(f"Attempt {attempt+1}: {status}")

        if status != "running":
            return status

        time.sleep(delay)

    raise Exception("Timeout waiting for crawl job")

This makes GET calls to the Cloudflare /crawl endpoint. It ensures you’re waiting for the task to complete processing before fetching the crawled records.

Tip: The limit=1 query parameter is recommended to restrict the number of retrieved records, keeping the response lightweight. After all, at this stage, you’re only interested in checking the job status, not in retrieving the actual output data.

Check the TWSC YouTube Channel

Step #4: Get the Crawled Content Pages

Build a fetch_records() function to collect all crawled pages:

def fetch_records(job_id):
    print("Fetching results...")

    all_records = []
    cursor = None

    while True:
        url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}"
        # Getting 10 records at a time
        params = {
            "limit": 10
        }

        if cursor:
            params["cursor"] = cursor

        response = requests.get(url, headers=HEADERS, params=params)
        response.raise_for_status()

        result = response.json()["result"]
        records = result.get("records", [])

        all_records.extend(records)
        print(f"+{len(records)} records (Total: {len(all_records)})")

        cursor = result.get("cursor")
        if not cursor:
            break

    return all_records

This handles pagination using a cursor, accessing records in batches (10 per request) until all results are returned.

Step #5: Put It All Together

Finally, in the main() function, orchestrate the workflow:

1. Start the crawl

2. Wait for completion

3. Fetch all results

Then, you can export the crawled records to a local JSON file for further use, store the retrieved data in a database, process it there, etc.

def main():
    job_id = start_crawl(START_URL)
    status = wait_for_completion(job_id)

    if status != "completed":
        raise Exception(f"Crawl failed: {status}")

    print("Crawl completed!")

    records = fetch_records(job_id)

    # Export the crawled pages to an output JSON file
    with open("records.json", "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

if __name__ == "__main__":
    main()

Step #6: Complete Code

This is what your Python script for interacting with the Cloudflare Crawl API will look like:

# pip install requests

import requests
import time
import json

# The Cloudflare secrets required for authenticating Crawl API calls
CLOUDFLARE_API_TOKEN = "<YOUR_CLOUDFLARE_API_TOKEN>"
CLOUDFLARE_ACCOUNT_ID = "<YOUR_CLOUDFLARE_ACCOUNT_ID>"

# The base URL used for all Crawl API calls
CLOUDFLARE_CRAWL_BASE_URL = f"https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/browser-rendering/crawl"

# Common headers shared by all API calls
HEADERS = {
    "Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}",
    "Content-Type": "application/json"
}

# The URL to crawl
START_URL = "http://quotes.toscrape.com/"

def start_crawl(start_url):
    """
    Triggers the Cloudflare Crawl API job
    """

    # Customize according to your needs
    payload = {
        "url": start_url,
        "limit": 20,
        "depth": 2,
        "formats": ["markdown"],
        "render": True
    }

    response = requests.post(CLOUDFLARE_CRAWL_BASE_URL, headers=HEADERS, json=payload)
    response.raise_for_status()

    job_id = response.json()["result"]
    return job_id

def wait_for_completion(job_id, max_attempts=60, delay=5):
    """
    Waits for the crawling task to complete
    """

    status_url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}?limit=1"

    for attempt in range(max_attempts):
        res = requests.get(status_url, headers=HEADERS)
        res.raise_for_status()

        status = res.json()["result"]["status"]
        print(f"Attempt {attempt+1}: {status}")

        if status != "running":
            return status

        time.sleep(delay)

    raise Exception("Timeout waiting for crawl job")

def fetch_records(job_id):
    """
    Collects all records from the paginated results
    """

    print("Fetching results...")

    all_records = []
    cursor = None

    while True:
        url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}"
        # Getting 10 records at a time
        params = {
            "limit": 10
        }

        if cursor:
            params["cursor"] = cursor

        response = requests.get(url, headers=HEADERS, params=params)
        response.raise_for_status()

        result = response.json()["result"]
        records = result.get("records", [])

        all_records.extend(records)
        print(f"+{len(records)} records (Total: {len(all_records)})")

        cursor = result.get("cursor")
        if not cursor:
            break

    return all_records

def main():
    job_id = start_crawl(START_URL)
    status = wait_for_completion(job_id)

    if status != "completed":
        raise Exception(f"Crawl failed: {status}")

    print("Crawl completed!")

    records = fetch_records(job_id)

    # Export the crawled pages to an output JSON file
    with open("records.json", "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

if __name__ == "__main__":
    main()

Step #7: Test the Script

Launch the script, and it’ll produce an output like this:

The polling mechanism required 5 attempts (~25 seconds), and the API discovered and retrieved 22 pages.

A records.json file will appear in your project directory. Open it, and you’ll see:

Notice how the “Quotes to Scrape” entries contain a markdown field with the Markdown version of the page. Instead, external links like Zyte’s homepage and Goodreads.com are skipped, since includeExternalLinks is set to false by default. In other words, the Cloudflare Crawl API doesn’t automatically attempt to fetch data from different domains than the target source URL.

Et voilà! Implementation complete.

Need help with your scraping project?

Benchmark Against Protected Websites

Cool! The Cloudflare Crawl endpoint works like a charm and is easy to use. However, I was particularly concerned about its documented limitations and wanted to verify whether they actually hold up in practice…

So, I ran tests against several well-known sites protected by common WAF and anti-bot solutions (from different providers). Here’s a summary of the results:

As you can tell, the limitations are very real. The results are quite discouraging: the Cloudflare Crawl API failed against all anti-bot–protected websites I tested.

So, is this solution reliable for web scraping? When (and how) should you actually use it? Let me break that down in a final comment!

Final Comment

In this article, I introduced you to one of the newest tools in Cloudflare’s growing ecosystem: the Crawl API! This endpoint is designed to help you crawl entire websites using distributed crawling tasks running on Cloudflare’s infrastructure.

Sure, the crawling mechanism works and is easy to launch, control, and implement. With just a few lines of code, you can get started. Still, several concerns should be raised:

1. Opaque pricing: Costs are tied to resource usage rather than the number of pages crawled, making them harder to predict.

2. Fixed User-Agent: The API doesn’t allow User-Agent customization, meaning even basic server-side checks can block it.

3. Limited effectiveness on protected sites: The API has an intended very low success rate against anti-bot–protected websites (unless you specify in Cloudflare Bot Protection settings that you allow it against your site).

4. Rate limiting constraints: It strictly respects robots.txt directives and crawl delays, which can significantly slow or limit large crawls.

In simple terms, if you want to use it for general-purpose, large-scale web crawling, I wouldn’t recommend it. The market offers more effective solutions that can actually bypass anti-bot limitations. Plus, remember that around 35% of the entire Internet is estimated to be protected against bots (i.e., you won’t be able to crawl it with this API).

Yet, if you know the target site is not protected, budget isn’t a concern, and you want to remain (overly?) ethical and compliant, the Cloudflare Crawl API can be an option.

I hope this breakdown helps you better understand this new solution and make an informed decision on whether to adopt it. Lastly, remember that the Cloudflare Crawl API is still in beta, so things may change soon. Just keep an eye on the docs for updates. Until next time!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

The short answer is that a couple of tools finished the task, just one with any consistency. The story behind why is worth telling, because it explains what is happening at the intersection of AI agents and web data right now. We ran a similar exercise against Cloudflare earlier this year, and the conclusion is broadly the same: each anti-bot needs its own answer, and the answer changes every quarter.

From workflows to agents, and why that changes the data problem

Most code shipped under the AI banner is not really agentic. It is workflow code with an LLM dropped into a slot: generate a summary here, classify a record there, draft an email at the end. The control flow is hard-coded, and the model is one component among many.

The definition of an agent is quite different. The model decides the next action, observes the outcome, and decides again. The control flow lives inside the loop, not outside it. The agent has goals rather than scripts, and it picks tools and steps based on what it sees. That is what makes the engineering interesting, that is what makes it hard, and that is what sometimes makes it unreliable.

It also forces a different relationship with data. An agent that only sees its training corpus is stuck in the past. To make decisions worth anything, it has to read prices that change daily, stocks that move minute by minute, listings that did not exist last week. Some of that data sits behind APIs. Most of it does not. The web is still the largest and most current dataset in the world, and most of it is reachable only through a browser. So if we want our agents to act on real information, we have to give them a way to browse: opening a page, reading it, clicking a link, typing into a search bar, following a result, filling a form, all on sites that were never built for machines.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

This is the constraint that produced the wave of “agentic browser” launches we have seen over the last twelve months. Y Combinator alone has backed a long string of them. Hyperbrowser (S21) was an early entry: scalable cloud browser infrastructure with built-in CAPTCHA solving, proxy management, and now a multi-agent playground. The newer cohort followed the agent wave more directly: BrowserOS (S24) is an open-source agentic browser that runs the agent locally on the user’s machine; Browser Use (W25) offers an open-source agent loop on top of Playwright, plus a cloud version. Skyvern is a self-hostable browser agent that uses an LLM and computer vision instead of fixed selectors. Outside the YC pipeline, Lightpanda is doing something different again, a headless browser engine written from scratch in Zig and aimed squarely at agents and crawlers (claiming roughly 9x faster execution and 16x lower memory than Chrome). It fits the “browser built for machines” line of thought we covered in Rethinking the web browser earlier this year. Browserbase ships a managed browser plus Stagehand for natural-language automation. And the big AI labs are now in the same space: OpenAI shipped Operator and the ChatGPT Atlas browser, Anthropic shipped Computer Use, Perplexity launched Comet. Each project attacks the same problem from a slightly different angle, but the goal is identical: a browser an agent can drive without immediately tripping every detection mechanism on the other side.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

The same problem scrapers have been chasing for a decade

For anyone who has worked in web data, none of this is new. The fight over whether a request looks human or automated has been going on as long as commercial scraping has existed. The product names have changed but the purpose not.

What has changed is who is selling the bypass. The companies that have spent years selling residential proxies and unblockers noticed quickly that the agentic boom is good for their business. They already have the IP networks, the fingerprint research, the bypass code, the cat-and-mouse experience. They know what TLS handshake Chrome sends in October 2025 and what it sent in October 2024. Pivoting all of that into a managed browser is a smaller leap than building one from scratch. Bright Data, Oxylabs, Rayobyte, ZenRows have all added a managed browser product alongside the proxy.

The other side of the line is moving in the opposite direction. Bot traffic has grown faster than human traffic for years, and the operators of large public sites care more about it than ever. DataDome, Cloudflare Bot Management, Akamai Bot Manager, HUMAN, Kasada: every one of them ships updates that target the exact tools we just listed. Fingerprint checks get stricter. Behavioral models get more sensitive. The JavaScript challenge changes shape every few weeks. There is no silver bullet, and there is no tool, browser, proxy, or service that bypasses every anti-bot on every site at all times. Anyone who claims otherwise is selling something that worked last quarter and might still work this week. The useful question is what works on a given target, today, at what cost.

Check the TWSC YouTube Channel

Picking a hard target

To answer that question concretely, we needed a target where the anti-bot was good and the signal was clean. We picked leroymerlin.fr, the French DIY retailer. Leroy Merlin runs DataDome standalone, with no other anti-bot layer on top, so attribution is straightforward. It also runs one of the more verbose DataDome configurations we have come across: response headers expose x-datadome-riskscore, x-datadome-protection, x-datadome-cid, and x-datadome-endpointid. Most DataDome-protected sites only show us the outcome. Here we see the score the engine assigns at every request, which is rare and very useful when comparing tools side by side.

The task we picked is small but realistic. From the homepage, the agent has to type “ampoule B22 led blanc” into the search bar, click the first product result, and add the product to the cart. Four steps. We dropped the login step on purpose: leroymerlin.fr requires an OTP to sign in, and we did not want OTP friction to confound an anti-bot test.

A run is a pass if the agent reaches the cart confirmation. Otherwise we record where it stopped and what DataDome said about it. Each tool runs ten times back to back, and we aggregate the results. Tools that support an external proxy use the same residential pool: Bright Data residential FR for the Bright Data runs, Geonode residential FR for the Geonode runs. Tools that ship their own proxy use it. The reason behind two different providers was because we wanted to diversify the IP addresses, to be sure that blocks were not a matter of IP reputation.

Need help with your scraping project?

The contestants

As you’ve seen before, the browser landscape is quite crowded and we could not cover all the tools. We picked four open-source projects and seven commercial products. Let’s start with the open source.

Camoufox is the stealth Firefox fork most people in the scraping world have already met (we introduced it on TWSC back in September 2024). It rotates real-world fingerprints, patches the obvious automation tells, and ships a Playwright-compatible API. We pair it with both Bright Data and Geonode residential proxies in France.

Pydoll takes a different route: it drives Chromium directly over CDP without WebDriver, with built-in humanized cursor movement and typing. Importantly, Pydoll implements an explicit Fetch.authRequired handler, which lets it authenticate proxies that require Basic auth.

Scrapling is a higher-level Python library. We use it in two modes. DynamicFetcher launches vanilla Playwright Chromium driven by Scrapling’s session manager. StealthyFetcher does the same, but under the hood uses an improved and customized version of patchright, a stealth-patched Playwright fork. Each gets its own row in the comparison.

RayoBrowse is the self-hosted stealth Chromium fork from Rayobyte, distributed as a Docker container that exposes a CDP endpoint on port 9222. Here we hit a wall worth flagging: for some reason RayoBrowse could not use the Bright Data residential proxy in our setup. Every navigation through that proxy failed instantly, even though the same credentials worked fine through curl from inside the same container. The same RayoBrowse setup worked fine with Geonode. We did not isolate the root cause, so we report RayoBrowse on Geonode only.

The commercial side is more crowded.

Browser Use exists in two flavors, and we tested both. The cloud version is the managed Browser Use, with its own residential proxy, its own stealth fingerprinting, and a fixed set of supported models; we drove it once in raw CDP mode (we steer it ourselves with Playwright) and once in agent mode (we hand the LLM the task in natural language and let it plan the steps).

Browserbase is a managed Chromium with optional residential proxies, Cloudflare Web Bot Auth verification, and the Stagehand agent SDK. We discovered during the test that the free tier excludes proxies entirely; without one, the session egresses from a US datacenter. We left this configuration in the test because it is what a free user would experience.

Browserless is a managed browser-as-a-service whose anti-bot story is a stealth path (/chromium/stealth) plus optional residential proxies for paid plans. The free plan caps sessions at 60 seconds, which is tight for a four-step flow. We tested it with the built-in residential proxy targeting France, and tried to test it with our external proxies via the externalProxyServer parameter; the external mode failed at connection time on every run, in the same Chromium-side authentication way that broke RayoBrowse, so we drop those configurations from the comparison.

ZenRows Scraping Browser is a managed Chromium with a built-in residential proxy network and built-in CAPTCHA solving; we connect via the WSS endpoint with proxy_country=fr to get a French exit point.

Bright Data Browser API sits at the other end of the same product category: a managed Chromium with built-in residential rotation and CAPTCHA solving, on a dedicated Browser API zone we configured on their dashboard.

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 103.BROWSERS.

What we had to fix before the numbers made sense

Read more

This means that if you route all your traffic through a single egress IP, you will be rate-limited in seconds and blacklisted in minutes. To survive at scale, you need to distribute your requests across a massive pool of IP addresses.

Traditionally, the web scraping industry has solved this issue thanks to commercial proxy providers. However, this is not the only approach. This article responds to the following question: “Is there a way to scrape at scale without burning budget on proxies?”

The answer is yes. But let’s be clear from the beginning: This approach is not a universal silver bullet. Let’s see how it works, how to build it, and what its limitations are.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

The Typical Solution for Scraping at Scale: Proxy Provider Services

Let’s start this discussion with the typical choice for scraping at scale. IP bans and rate limits are the #1 operational problem in scraping, especially at scale. The typical solution every web scraping engineer integrates is using proxy servers, for a simple reason: proxies act as intermediaries between your scrapers and the Internet, avoiding your scrapers from getting banned. To do so, companies buy proxy IPs from proxy providers. The most common categories, both with their flaws, are the following:

• Datacenter proxies: These are cheap and fast, but their ASNs(Autonomous System Numbers) are heavily scrutinized. WAFs maintain databases of known datacenter CIDR (Classless Inter-Domain Routing) blocks, so hitting a target with a static list of 100 datacenter proxies usually results in those IPs being flagged and blocked within hours.

• Residential proxies: These route traffic through actual consumer devices. They have highly trusted IP reputations, making them excellent for bypassing anti-bot systems. However, they are priced by bandwidth, so they are very expensive, especially when scraping at scale.

The main limitation of this approach is that it is highly expensive. So, what if you need to scrape at scale but don’t have enough budget for doing so?

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

An Alternative Approach: Scraping at Scale With Dedicated Infrastructure

To escape metered billing, you can move egress back to dedicated infrastructure. But before presenting the solution, let’s first point out shortly what happens when you buy and use proxies, at the infrastructure level.

Buying Proxies Means Delegating Your Infrastructure

When you buy proxies from providers, you are delegating 100% of your infrastructure. When your scrapers make the requests, under the hood, the proxy provider connects to a gateway, which is a massive load balancer controlled entirely by the provider itself.

Let’s consider the case of residential proxies, for simplicity. Behind the gateway is a peer-to-peer (P2P) network of millions of consumer devices that the provider has acquired bandwidth from. When your request hits the gateway, their proprietary routing algorithm decides which consumer device in which country will act as your final exit node.

The second you route traffic through their gateway is the exact moment where you delegate the 100% of your scraping infrastructure.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

NyxProxy: The Infrastructural Solution

NyxProxy is a self-hosted HTTP/SOCKS5 proxy server that exploits a well-known IPv6 networking trick: When a cloud provider gives you a /64 subnet, you legally own 18.4 quintillion IPv6 addresses.

Let’s explain the number and the trick around IPv6s. An IPv6 address looks like this:

 2a05:f480:1800:25db:0000:0000:0000:0001

They are 128 bits long. That gives 2^128 possible addresses. The number is so large that the designers said: “We can afford to give every organization a massive block and never worry about running out”.

Now, here is the trick. An IPv6 address is split into two halves, 64 bits each:

2a05:f480:1800:25db : 0000:0000:0000:0001
|___________________|   |_________________|
   Network prefix            Host part
   (your subnet)          (you control this)

The /64 notation means: the first 64 bits identify the network, the last 64 bits are yours to assign however you want. The last 64 bits can be any value from 0000:0000:0000:0000 to ffff:ffff:ffff:ffff: That’s 2^64 = 18.4 quintillion combinations. All valid addresses, all routable to your server.

Thanks to this trick, NyxProxy can assign a pool of those addresses to your network interface at startup, then rotate your outgoing traffic across them. This means having a fresh IP per request. The tool handles pool management, background rotation, NDP proxying via ndppd, and exposes a monitoring endpoint.

The best part is, indeed, in the NDP proxying. When your server uses a random address like 2a05:f480:1800:25db:a3f1:9922:beef:1234 as a source IP, your router upstream needs to know your server is responsible for that address. Otherwise, the response packets have nowhere to go.

IPv6 uses NDP (Neighbor Discovery Protocol) for this. The router sends an NDP query: “who has 2a05:f480:1800:25db:a3f1:9922:beef:1234?” and your server must answer.

ndppd (NDP Proxy Daemon) runs on your server and answers those queries automatically for your entire /64 subnet, essentially saying “yes, all of those addresses are mine”. Without it, your packets go out, but responses never come back.

Below is a summary schema of how this whole process works:

Provider gives you:  2a05:f480:1800:25db::/64
                     ↓
Your server can use: 2a05:f480:1800:25db:[anything]
                     ↓
NyxProxy assigns 200 random IPs to your interface
                     ↓
Each outgoing request binds to a different one
                     ↓
Target sees 200 different source IPs
                     ↓
ndppd makes sure responses route back correctly

Check the TWSC YouTube Channel

How To Use NyxProxy

Let’s now see how to use NyxProxy with a practical implementation.

Environment Setup & Prerequisites

To replicate this tutorial for deploying NyxProxy and utilizing it in your scraping scripts, you must have the following system and hardware requirements:

• Hardware: A Virtual Private Server (VPS) or bare-metal server with at least 512 MB of RAM and 100 MB of disk space. Supported architectures are amd64 or arm64.

• Subnet: A cloud provider that natively delegates a full IPv6 /64 subnet to your network interface. Note that not all the VPS providers are supported: Check out the NyxProxy documentation to learn more about supported VPSs.

• Operating system: A modern Linux distribution, specifically Ubuntu or Debian, to ensure compatibility with the automated setup scripts and sysctl kernel modifications.

• Python: Python 3.7 or higher installed on your local machine to run the scraping scripts.

To get your server ready to run the proxy daemon, you need to verify your IPv6 setup and gain root access. Ensure you are logged into your VPS via SSH as the root user, or have sudo privileges.

First, verify that your server has a globally routable IPv6 /64 subnet assigned to it. You can check this by running the following command in your server’s terminal:

ip -6 addr show | grep "scope global"

If done correctly, you should see an output similar to the following:

inet6 2a05:f480:1800:25db::1/64 scope global

If you do not see a /64 subnet, you will not be able to rotate IPs, and you must review your cloud provider’s network settings.

Next, prepare your local development environment. Suppose you call the main folder of your Python project nyxproxy_scraper/. At the end of this step, the folder will have the following structure:

nyxproxy_scraper/
    ├── main.py
    └── venv/

Where:

• main.py is the Python file that will store your proxy request logic.

• venv/ contains the standard Python virtual environment.

You can create the venv/ virtual environment directory like so:

python -m venv venv

To activate it, on Windows, run:

venv\Scripts\activate

Equivalently, on macOS and Linux, execute:

source venv/bin/activate

As a final prerequisite, install the Requests library in your activated virtual environment so your Python script can make HTTP calls:

pip install requests

Well done! You are now ready to test and use Nyxproxy.

Need help with your scraping project?

Installing and Configuring NyxProxy

NyxProxy provides a quick setup script that handles the infrastructural heavy lifting. It auto-detects your network interface, installs ndppd, tweaks the Linux kernel parameters via sysctl to allow non-local binding, and downloads the compiled Go binary.

You can launch it with the following single command:

wget <https://raw.githubusercontent.com/jannik-schroeder/nyxproxy-oss/main/scripts/quick-setup.sh> && chmod +x quick-setup.sh && sudo ./quick-setup.sh

During the setup, you will be prompted to configure your proxy credentials and set your rotation rules. Behind the scenes, the script generates a config.yaml file. Let’s look at the crucial subset of that configuration:

network:
  rotate_ipv6: true
  ipv6_subnet: "2a05:f480:1800:25db::/64"

  # The rotation mechanics:
  ipv6_pool_size: 200
  ipv6_max_usage: 100
  ipv6_max_age: 30

Below is an explanation of what these three parameters mean for your scraping pipeline:

• ipv6_pool_size: NyxProxy keeps 200 mathematically unique IPs “hot” and bound to your network interface at any given time. This keeps proxy startup times under 100ms while maintaining IP diversity.

• ipv6_max_usage: After a specific IP has been utilized for 100 requests, it is considered “burned.” NyxProxy destroys the route and spins up a fresh address to dynamically replace it.

• ipv6_max_age: If an IP hasn’t hit 100 requests but has been alive for 30 minutes, it gets forcefully rotated out. This prevents time-based algorithmic tracking by the target WAF.

Once the daemon is running as a systemd service, your VPS is officially acting as a rotating proxy gateway. When NyxProxy receives a scraper request, the underlying Go binary takes over. It looks at its internal memory, picks one of the 200 rotating IPv6 addresses in its pool, and binds to that specific address to establish the outbound connection.

The expected output is as follows:

IPv6 rotation mode: IP Pool with dynamic rotation
  Interface: enp1s0
  Subnet: 2a05:f480:1800:25db::/64
  Pool size: 200 IPs
  Rotation: Every 100 uses or 30m0s
  Initializing IP pool...
  Progress: 50/200 IPs added
  Progress: 100/200 IPs added
  Progress: 150/200 IPs added
  Progress: 200/200 IPs added
  IP pool ready with 200 addresses
  Background IP rotation started

Starting https proxy on 0.0.0.0:8080 (Protocol: IPv6)

Testing the Proxy Logic

At this point, NyxProxy has done its job. To verify it works correctly, you can use the following Python script that hits api6.ipify.org, which is an API that simply bounces back the IP address it sees:

import requests

# Point this to your VPS IP and the credentials you set during setup
proxies = {
    'http': '<http://admin:password@your-vps-ip:8080>',
    'https': '<http://admin:password@your-vps-ip:8080>'
}

# Test 5 consecutive scraping requests
for i in range(5):
    response = requests.get('<https://api6.ipify.org>', proxies=proxies)
    print(f"Request {i+1}: Target sees IP -> {response.text}")

(NOTE: If you are already familiar with ipify.org, note that the “api6” prefix can be used for IPv6 requests only.)

The result should be similar to the following:

Request 1: Target sees IP -> 2a05:f480:1800:25db:1a2b:3c4d:5e6f:7890
Request 2: Target sees IP -> 2a05:f480:1800:25db:9988:7766:5544:3322
Request 3: Target sees IP -> 2a05:f480:1800:25db:aaaa:bbbb:cccc:dddd
Request 4: Target sees IP -> 2a05:f480:1800:25db:1122:3344:5566:7788
Request 5: Target sees IP -> 2a05:f480:1800:25db:dead:beef:cafe:babe

This shows that every single HTTP request utilizes a completely different, globally routable IPv6 address generated from your subnet block. To the target server, these look like entirely distinct users connecting from across the internet.

Perfect! You have successfully built a self-healing, infinitely rotating proxy pool without handing over your budget for metered residential bandwidth.

The Illusion of Infinity: Critical Limitations of IPv6 Subnet Rotation

At this point, you may think you have found a solution to all of your budgeting problems for scraping at scale. But before you tear down your commercial proxy infrastructure, you must understand that a $5/Mo VPS and an open-source rotation daemon are not a universal silver bullet. If it were that simple, the commercial proxy industry would not exist.

This architecture has the following main limitation:

• The IPv4 compatibility wall: This entire architecture is built on one absolute prerequisite: Your target endpoint must support IPv6. If you are scraping legacy enterprise systems or platforms that haven’t migrated to dual-stack networking, this setup is useless. You cannot route an IPv6 packet to an IPv4-only server.

• Subnet-level bans (/64 prefix blocking): Enterprise WAFs are fully aware of IPv6 prefix delegation standards. They know that hosting providers allocate a /64 subnet to a single client. If their heuristics detect highly concurrent behavioral patterns (like missing browser fingerprints or anomalous TLS handshakes) originating from 2a05:f480...:1a2b, they will ban the entire /64 CIDR block. Once your /64 prefix is banned, all 18 quintillion of your “infinite” IPs are simultaneously dead. To recover, you must physically destroy the VPS and provision a new one in a different IP range.

• ASN reputation: No matter how many IPs you rotate, your traffic still originates from a Datacenter Autonomous System Number (ASN). Target firewalls assign a baseline trust score to every ASN. Traffic originating from a Datacenter ASN always starts with a highly degraded trust score compared to a Residential ASN. For highly restrictive targets, any request from a datacenter IP is instantly met with an unpassable CAPTCHA or a hard 403 Forbidden, regardless of whether it’s IPv4 or IPv6.

• nf_conntrack and hardware exhaustion: You cannot push enterprise-grade throughput on a $5, 1-vCPU server without consequence. Rotating thousands of IPv6 addresses requires the Linux kernel to aggressively maintain the nf_conntrack table and the NDP proxy table. At high concurrencies, the overhead of establishing, tracking, and tearing down thousands of TCP sockets across rotating interfaces will exhaust the memory or CPU of a low-tier VPS. The kernel will begin dropping packets natively, your latency will spike to useless levels, and your scrapers will be greeted with errors.

Conclusion

In this article, you learned how to leverage your hosting provider’s IPv6 /64 subnets to build an infinitely rotating proxy pool with NyxProxy, escaping the metered billing of residential proxy networks.

The competitive advantage of engineering your own proxy infrastructure is in your unit economics and architectural control. However, you also learned that this solution is not a universal silver bullet for every scraping scenario: It comes with trade-offs and constraints.

So, let us know: Have you already experimented with bare-metal IPv6 rotation for your scraping pipelines? What targets did it work best for? Let’s discuss in the comments!

Behind the scenes, it relies on hydration to make server-rendered pages interactive. And here’s the interesting part: the same mechanism that makes Next.js fast and popular also exposes a significant amount of structured data in the HTML sent by the server. From a scraping perspective, that’s a huge opportunity!

In this post, I’ll show you a simple trick to scrape data from virtually any Next.js website. Follow along as I break down how it works and how you can apply it yourself.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Next.js in Numbers

Next.js needs no introduction, but it’s worth giving some context to truly understand how popular it is (and therefore how useful the trick I’m about to present for Next.js web scraping can be):

• According to the 2025 Stack Overflow Developer Survey, 20.8% of respondents used Next.js extensively over the past year.

• Next.js is the 14th largest repository on GitHub, with over 138k stars (and still growing!).

• According to W3Techs, Next.js has a 2.9% market share among JavaScript libraries.

• Major brands such as Nike, Stripe, and Notion have chosen this full-stack framework to build their official websites.

Before Getting Started: A Bit of Context on Hydration

I know you probably just want the trick… Still, let me take a minute to explain why it works in the first place, why it’s even possible, and what kind of data you’ll actually retrieve with it!

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

What Is Hydration?

Hydration is the process that makes a server-rendered page interactive in the browser.

Frameworks like Next.js, Remix, Nuxt, and SvelteKit employ this mechanism to combine the performance benefits of server-side rendering (SSR) with the interactivity of client-side applications.

The idea is that the server first sends fully rendered static HTML to the browser. Then, hydration happens next.

The browser downloads the JavaScript bundle, and the frontend framework reconstructs the component tree in memory, attaches event listeners, and links that virtual tree to the existing DOM instead of re-rendering it from scratch. The result is a fully interactive application built on top of server-rendered HTML.

How Does the Hydration Mechanism Work?

It’s now clear that in Next.js and similar frameworks, hydration is the process where a static, server-rendered HTML page “comes to life” and becomes fully interactive in the browser. But what’s actually happening under the hood?

At a high level, hydration is a 3-step process:

1. The server generates and sends a fully rendered HTML snapshot. The user immediately sees the content (great for First Contentful Paint). At this point, though, the page is just static HTML. Buttons, forms, and other interactive elements are visible, but they don’t work yet because no JavaScript is attached.

2. The client’s browser downloads the JavaScript bundle (which includes React and your frontend application code) and executes it.

3. React rebuilds the component tree in memory and attaches event listeners to the existing DOM nodes. Instead of discarding the HTML and re-rendering everything from scratch, React “hydrates” the existing markup, meaning it reuses it and wires it up with state and interactivity.

Once hydration completes, the page behaves like a normal single-page application: it responds to clicks, manages state, and updates dynamically.

And here’s an important detail: if the browser doesn’t support JavaScript (or it fails to load), the user still sees the server-rendered HTML. It won’t be interactive, but the core content is there. That’s great for SEO and perceived performance!

Why It Matters for Scraping Next.js (and Other Full-Stack Frameworks…)

The key insight you need to understand is simple: hydration requires data, and that data must be embedded somewhere in the HTML sent by the server!

In Next.js, when the server renders a page, it doesn’t only send markup. It also serializes the data required to rebuild the React component tree on the client. That serialized payload is embedded directly into the page’s HTML.

That’s exactly why hydration matters for scraping. Instead of parsing the DOM or simulating user interactions through browser automation, you can extract the structured data that React itself uses to hydrate the page.

In many cases, hydration data is cleaner and easier to parse than the rendered HTML. It can also contain more information than what’s visibly displayed on the page, including hidden and interesting metadata.

Keep in mind that this principle applies not only to Next.js! All other full-stack frameworks that rely on hydration, such as Remix, Nuxt, Angular Universal, and SvelteKit, tend to dehydrate state on the server and rehydrate it on the client.

So remember this simple rule. If a framework hydrates, it must serialize data. And if it serializes data into the HTML, you can scrape it.

How to Scrape Next.js Websites: 2 Approaches

The approach to scraping Next.js by targeting hydration data depends on how that data is embedded in the HTML generated on the server side.

I won’t go too deep into framework internals here (if you’re a Next.js dev, you already know things shift depending on whether you’re using the App Router or the Pages Router), but there are essentially two scenarios you’ll run into.

In this section, I’ll walk through both of them and show you exactly how I retrieve data from each!

Check the TWSC YouTube Channel

Approach #1: Target the __NEXT_DATA__ Script

As a target site, I’ll use a Nike product page as a reference:

That’s actually a great example because Nike.com is even showcased on the Next.js homepage as a real-world site built with the framework.

Now, right-click on the page and select the “Inspect” option in your browser to open the DevTools. Scroll through the DOM and get familiar with the page structure. If the Next.js site is using the Pages Router, you’ll notice a <script> tag with the id __NEXT_DATA__ containing a large JSON blob:

That JSON data is precisely the hydration data I was referring to earlier.

When a site uses the Pages Router approach in Next.js, the server embeds all the page data directly into that <script> tag. From a scraping perspective, that’s gold, as the data is already structured and ready to be captured.

Below’s a simple JavaScript snippet to extract it:

const hydartionScript = document.querySelector("#__NEXT_DATA__")
const hydrationData = JSON.parse(hydartionScript.innerHTML)
console.log(hydrationData)

What’s happening here is straightforward. The JS script:

• Selects the <script> element with id __NEXT_DATA__.

• Reads its inner HTML (which is a JSON string).

• Parses it into a JavaScript object.

• Logs it to the console.

Run this directly in the DevTools Console, and you’ll immediately see the result:

What’s interesting is how much structured data you get right away. This includes product details, images, metadata, and more. All is neatly organized, and it only took three lines of code!

If you want to store the JSON hydration object, just right-click the object in the Console and select the “Copy object” option:

From there, you can paste it wherever you need (e.g., into a local .json file, a MongoDB collection, etc.).

Need help with your scraping project?

Approach #2: Target the self.__next_f.push() Elements

Another, more complex approach to scraping Next.js involves pages built with the App Router.

Even if the App Router has been the recommended direction for a while, in my experience, it’s still not as widely adopted as the Pages Router. And honestly, that’s a bit of a gift for us (as scraping hydration data in App Router sites is definitely more complex!)

As a reference, let’s look at the “Business Overview” page on the OpenAI website, which is built with Next.js App Router:

Just like before, open DevTools and inspect the page. This time, focus on the <script> tags inside the <body>:

You’ll notice several scripts containing content like:

self.__next_f.push(<some_data>)

That “<some_data>” is serialized using the React Flight protocol for React Server Components (RSC). I won’t go too deep into the internals here (it’s a dense topic!), but what matters is that deserializing that data is not straightforward!

React Flight isn’t plain JSON. It mixes control records (HL, I, J, etc.), module references, streaming boundaries, and serialized model fragments into a transport format that React incrementally resolves at runtime.

You might think: “Why not just reuse the frontend deserialization library?” In practice, that doesn’t work well because:

• The client decoder (react-server-dom-webpack) expects a full React runtime.

• It relies on module maps and webpack IDs generated at build time.

• It resolves component references against the exact bundle that produced the stream.

• It assumes streaming semantics and internal React wiring.

Basically, outside that exact environment, you don’t have the module graph, build manifest, or hydration context. So even if you import the decoder, you can’t reconstruct the component tree the way the browser does.

There have been recent security issues in the React Flight payload deserialization system, highlighting just how sensitive and complex this layer is. For more details, refer to:

• Security Advisory: CVE-2025-66478

• Critical Security Vulnerability in React Server Components

Thus, instead of fighting the protocol, I’d simplify and accept that in this case, it’s better to extract the unparsed React Flight string data. Achieve that with the JS script below:

const nextFlightScripts = [...document.querySelectorAll("script")]
  .filter(script => script.textContent.includes("self.__next_f"))
  .map(script => script.textContent.trim())
console.log(nextFlightScripts)

This selects all <script> elements containing “self.__next_f” and builds an array of their raw contents.

Run it in the Console, and you’ll get an array of React Flight chunks:

From there, the simplest way to extract structured data is often to copy the array, feed it to an AI, and ask it to reconstruct a parsed JSON representation of the meaningful payload sections:

Is this more complicated than the __NEXT_DATA__ trick? Absolutely! Yet, it’s still a powerful way to access a large amount of page data with just a few lines of code.

Final Script to Quickly Access Data From Next.js Sites

If you combine the two approaches, you can build a production-ready script for brute-force hydration data scraping in Next.js:

// Pages Router approach (__NEXT_DATA__)
const hydrationScript = document.querySelector("#__NEXT_DATA__")
let nextData = null
if (hydrationScript) {
  try {
    nextData = JSON.parse(hydrationScript.textContent)
    console.log("__NEXT_DATA__ found:")
    console.log(nextData)
  } catch (err) {
    console.warn("Failed to parse __NEXT_DATA__:", err)
  }
} else {
  console.log("No __NEXT_DATA__ script found.")
}

// App Router approach (self.__next_f)
const nextFlightScripts = [...document.querySelectorAll("script")]
  .map(script => script.textContent.trim())
  .filter(content => content.includes("self.__next_f.push"))

if (nextFlightScripts.length > 0) {
  console.log("React Flight scripts found:")
  console.log(nextFlightScripts)
} else {
  console.log("No React Flight scripts found.")
}

To test it, just open the Console in DevTools, paste the script, and run it.

Important: The <script> components containing hydration data aren’t loaded dynamically via client-side rendering. They’re embedded directly in the HTML generated by the server.

That means you can:

1. Fetch the target Next.js-powered page with an HTTP client.

2. Parse the HTML using an HTML parsing library like Beautiful Soup or Cheerio.

3. Apply a similar version of the JavaScript script above, but adapt it to the API provided by your HTML parser.

In other words, this trick for scraping Next.js doesn’t only work in the browser DevTools. It also works perfectly in regular scraping scripts!

Pros and Cons of This Approach to Next.js Scraping

👍 Pros:

• Simple and effective, requiring only a few lines of code.

• Works on all Next.js websites (and, more generally, on most sites that rely on hydration).

• Can let you access more data than what is actually displayed on the page.

• No need for browser automation, waiting for client-side rendering, or simulating user interactions.

👎 Cons:

• You may only get partial data, meaning you might still need to complement it with a more traditional scraping approach.

• React Flight data is difficult to parse and may require custom logic or even AI-assisted parsing.

Conclusion

Here, I’ve shared a trick I personally documented years ago, and that still works to this day. It allows you to quickly scrape data from virtually any Next.js site by targeting the hydration data embedded in the HTML document generated by the server and sent to the client for rendering.

As you’ve seen, with just a few lines of JavaScript, you can extract hydration data from any Next.js-powered page. What you get back is clean, or at least almost clean, data that you can process directly in your data pipelines.

Instead of fighting the frontend, this Next.js web scraping approach helps you leverage the data the framework itself needs to function!

I hope you found this useful and insightful. If you have questions or thoughts, feel free to share them in the comments below!