That’s where the MarkItDown Python library comes into play!

Here, I’ll show you everything you need to know about MarkItDown, from installation to practical examples, so you can convert HTML pages, PDFs, Word documents, Excel spreadsheets, and many other data sources into LLM-ready Markdown.

Why the Markdown Data Format Is More Relevant Than Ever

As I covered in a recent post, AI-ready web scrapers, RAG pipelines, and agent frameworks all have one thing in common: they rely heavily on Markdown!

That’s no coincidence... Mainstream LLMs (including OpenAI’s latest models) natively “speak” Markdown and often generate Markdown-formatted responses without being prompted.

Why? Because Markdown strikes the perfect balance between plain text and structured documents. It preserves headings, lists, tables, links, code blocks, and other semantic elements without the overhead and noise of HTML.

As a result, LLMs can understand and process Markdown efficiently while consuming far fewer tokens. This translates directly into lower inference costs and leaves more of the context window available for meaningful content.

In my experience (and as I demonstrated in a benchmark I conducted for Bright Data), Markdown has become the natural format for feeding documents, especially content extracted from the web, into LLMs.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

---

Why Markdown Conversion Matters in Web Scraping Pipelines

Web data is rarely the final destination. Quite the opposite, scraped content generally represents the starting point of a larger data analysis pipeline, where the goal is to extract insights, enrich datasets, or power downstream applications.

In these workflows, the information you need tends to go beyond traditional web pages. PDFs, Word documents, videos, presentations, and other files can provide additional context required for accurate analysis and processing. These resources may come from the web and be collected during scraping, or they may be added later.

Regardless of the original format, data is increasingly processed by AI systems. Since LLMs naturally work well with Markdown, converting different sources into a clean, structured Markdown representation can greatly simplify your data pipelines.

That’s why you need a multi-format Markdown conversion library in your toolkit, such as MarkItDown!

The new anti-bot solution - Toughest walls, lowest price. Claim your free 10 000 requests with coupon code WSCLUB

Claim your credits

MarkItDown: A Complete Overview

Let me now introduce you to MarkItDown, one of the most popular libraries for converting multiple input formats into Markdown.

What Is MarkItDown?

MarkItDown is an open-source Python library for converting a wide range of documents and other data sources into Markdown. The project is developed and maintained by Microsoft.

It’s quickly become one of the most popular tools in its category (more on this later in the post), with over 165k GitHub stars, over 12 million monthly downloads, and a thriving community.

MarkItDown supports dozens of input formats, ranging from Office documents and PDFs to images, audio files, web pages, YouTube videos, and structured data formats such as JSON, XML, and CSV.

Key Features

The main capabilities supported by MarkItDown are:

• Broad format support: Converts dozens of input formats, including PDFs, Office documents, HTML, images, audio, EPUBs, ZIP archives, YouTube URLs, and structured data files.

• Python API: Offers a clean, developer-friendly API for integrating document conversion into Python applications.

• Native CLI: Includes a rich CLI for converting files directly from the terminal, making it ideal for scripting, batch processing, and DevOps workflows.

• Plugin architecture: Supports third-party plugins, allowing the community to extend the library with custom converters and additional processing features.

• AI-powered processing: Integrates with LLMs to generate image descriptions, perform OCR, and transcribe audio, enriching the resulting Markdown.

• Flexible dependency management: Lets you install only the converters you need, reducing dependencies and keeping deployments lightweight.

• MCP server support: Provides a lightweight Model Context Protocol server that allows AI agents to call MarkItDown as a tool.

• Azure AI integration: Works with Azure Document Intelligence and Azure Content Understanding for higher-quality OCR, layout analysis, and structured field extraction.

Supported Input Formats

Installation and Setup

Follow the steps below to add MarkItDown to your system.

Prerequisites

MarkItDown requires Python 3.10 or higher. It’s also recommended to install it inside a Python virtual environment.

Installation via pip

The easiest way to install MarkItDown is through the PyPI markitdown package with:

pip install 'markitdown[all]'

Note that [all] option adds all optional dependencies required to support every available input format.

If you only need support for specific file types, you can install the specific dependencies individually for more control. For example:

pip install 'markitdown[pdf, docx]'

This adds only the extra dependencies required for PDF and Word file conversion.

Available optional dependencies include:

• [all]: Installs all optional dependencies.

• [pptx]: Adds support for PowerPoint files.

• [docx]: Adds support for Word documents.

• [xlsx]: Adds support for Excel files.

• [xls]: Adds support for older Excel files.

• [pdf]: Adds support for PDF documents.

• [outlook]: Adds support for Outlook messages.

• [audio-transcription]: Adds support for WAV and MP3 audio transcription.

• [youtube-transcription]: Adds support for YouTube video transcription.

• [az-doc-intel]: Enables integration with Azure Document Intelligence.

• [az-content-understanding]: Enables integration with Azure Content Understanding.

Note: For installation from source or Docker-based usage, refer to the official MarkItDown docs.

MarkItDown in Action: Full Examples

Time to explore some practical examples of converting different input data formats into Markdown.

HTML Document to Markdown

Suppose you have an HTML page stored locally (e.g., as part of an offline web scraping procedure). You can convert it to Markdown using the MarkItDown CLI:

markitdown example.html -o example.md

Alternatively, you can achieve the same result via the Python API:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("example.html")

with open("example.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

Now, assume the input is the following HTML page (i.e., the homepage from example.com):

The resulting example.md file will contain:

Notice how MarkItDown focuses on the meaningful HTML content, ignoring tags like <style>.

The library extracts the relevant elements and converts them into a structured Markdown representation, keeping important information such as headings, links, and text formatting while removing unnecessary HTML overhead.

Web Page to Markdown

A particularly useful feature for web scraping workflows is that MarkItDown can also fetch and convert web pages directly from their URLs, eliminating the need to manually save the HTML first.

Just pass the URL of the target page directly to the MarkItDown CLI:

markitdown 'https://example.com' -o example.md

Or to its Python API:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("https://example.com")

with open("example.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

Under the hood, MarkItDown performs an HTTP GET request to the specified URL using Requests, retrieves the page HTML, and converts the resulting content into Markdown. As usual, keep in mind that the request may fail if the target website uses anti-bot protections.

The output will be the same example.md Markdown file shown in the previous example.

Check the TWSC YouTube Channel

PDF to Markdown

Many websites store valuable information in downloadable PDF files (e.g., shopping flyers, business reports, whitepapers, product documentation, etc.).

While you can send these files directly to an AI model, doing so often consumes unnecessary context and tokens because PDFs may contain complex layouts and additional formatting overhead.

MarkItDown converts PDFs into clean Markdown while preserving important elements such as headings, document structure, lists, and tables, making the content easier for LLMs to process.

Assume your input is the sample PDF below:

Convert it to Markdown with the MarkItDown CLI:

markitdown sample.pdf -o sample.md

Or in Python:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("sample.pdf")

with open("sample.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

The result will be this sample.md file:

As you can see, MarkItDown preserves all the meaningful content from the PDF.

Excel to Markdown

Now, suppose you have relevant internal data stored in an Excel spreadsheet and want to pass it to an LLM for analysis or processing:

Convert it to Markdown through the MarkItDown CLI:

markitdown spreadsheet.xls -o spreadsheet.md

Or use the Python API:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("spreadsheet.xls")

with open("spreadsheet.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

The result will be a Markdown file containing a table with the same data as the original spreadsheet:

Or, for a better visual representation, through the Markdown viewer:

Need help with your scraping project?

MarkItDown Plugins

MarkItDown also supports third-party plugins that extend its built-in capabilities with additional converters and processing features. You can discover available plugins by searching GitHub for the #markitdown-plugin hashtag:

To list all installed plugins, run:

markitdown --list-plugins

Plugins are disabled by default. To enable them from the CLI, add the --use-plugins flag:

markitdown --use-plugins <SOURCE_FILE_PATH> -o <OUTPUT_FILE>

If you’re using the Python API, enable plugins by setting the enable_plugins parameter to True:

from markitdown import MarkItDown

md = MarkItDown(
  enable_plugins=True,
)

Once enabled, MarkItDown automatically loads all installed plugins.

MarkItDown Alternatives: Other Document-to-Markdown Libraries

MarkItDown isn’t the only library for converting documents to Markdown. Other popular libraries for converting different types of data into Markdown include:

Conclusion

Markdown is the language of LLMs and, as a result, the preferred format for AI-powered data processing and analysis pipelines. In this post, I covered how to use MarkItDown to convert web pages, HTML documents, PDFs, Word documents, Excel spreadsheets, PowerPoint presentations, YouTube videos, and many other file formats into clean, AI-friendly Markdown.

You now know how to get the most out of MarkItDown to prepare data for AI workflows, regardless of whether it comes from web scraping, downloaded files, or local documents.

I hope you found this guide useful. If you have any questions or comments, feel free to leave them below. Thanks for reading, and see you in the next one!

FAQ

How to set a proxy in MarkItDown?

MarkItDown doesn’t provide a dedicated proxy configuration option. Still, it relies on the Python requests library internally, which supports proxy configuration through special environment variables. Thus, you can set a proxy in MarkItDown with:

export HTTP_PROXY="http://user:password@your-proxy-server.com:port"
export HTTPS_PROXY="http://user:password@your-proxy-server.com:port"

Or, directly in the Python code with:

import os
from markitdown import MarkItDown

# Set your proxy addresses (replace with your proxy details)
os.environ["HTTP_PROXY"] = "http://user:password@your-proxy-server.com:port"
os.environ["HTTPS_PROXY"] = "http://user:password@your-proxy-server.com:port"

md = MarkItDown()
# ...

How to deal with 429 errors when converting YouTube videos to Markdown with MarkItDown?

Try to convert a YouTube video to Markdown, such as with:

markitdown "https://www.youtube.com/watch?v=1wtQAoBZLTg" -o video.md

The request is likely to fail with 429 Too Many Requests errors due to YouTube anti-bot protections. This can happen even with a fresh IP address. To improve reliability, try to route MarkItDown requests through a high-quality residential proxy.

How to use the MarkItDown MCP server?

The MarkItDown MCP server exposes a single tool that allows AI agents and MCP-compatible applications to convert different data sources into Markdown. The tool is convert_to_markdown, which accepts http:, https:, file:, and data: URIs.

Install the MarkItDown MCP server with:

pip install markitdown-mcp

Then, connect it to an MCP-compatible client, such as Claude Code, Claude Desktop, Codex, Gemini CLI, etc.

As AI usage is growing more and more every day, you’d like an AI-powered tool that can improve your scraping productivity, don’t you?! Especially when it comes to managing selectors, and DOM structure changes…

Well, there’s good news for you: this is the exact gap Bright Data Scraper Studio is designed to close!

This article discusses what Bright Data’s Scraper Studio is, why it exists, and how to use it with a step-by-step tutorial.

Let’s get into it!

What Scraper Studio Actually Is

Scraper Studio is a cloud-hosted, end-to-end scraping environment running entirely on top of Bright Data’s infrastructure. The tool is designed from the ground up with scraping primitives in mind, as it provides built-in proxy rotation, native JavaScript rendering, anti-bot handling, and direct integrations with data delivery targets like Amazon S3, Snowflake, Google Cloud, and Microsoft Azure.

The platform offers two build modes that sit on the same underlying scraper runtime:

• AI agent mode: You provide a target URL and a plain-English description of the data you want to extract. The AI agent visits the page, infers a structured output schema, and then generates the full scraper code for you. At the end of this automated process, you review and approve the schema before code generation begins. This gives you a checkpoint to catch misaligned fields before they propagate into your pipeline, allowing you to change them.

• Full IDE mode: A full browser-based JavaScript code editor with debugging tools for when you need direct, line-level control over the scraper logic. This is the mode you can use for when you’re dealing with multi-stage scrapers, custom pagination logic, complex state management between page navigations, or sites with particularly aggressive anti-bot stacks that need fine-tuned handling.

What’s more important is that both modes produce the same type of scraper. This means that you’re never locked into one approach: a scraper built in the AI Agent mode can be opened and edited in the IDE at any time, and any scraper can be updated using the Self-healing tool.

Scraper Studio Key Features

Let’s take a close look at Scraper Studio’s features:

• Built-in proxy infrastructure: Scraper Studio runs on Bright Data’s proxy network which, among other types, provides users with 400+ million residential IPs across 195 countries, with 99.99% uptime. For scraping professionals, this means no proxy pool management and, more importantly, no dealing with burned IPs.

• Anti-bot bypass, natively integrated: JavaScript rendering, fingerprinting evasion, and bot detection bypass are built into the runtime. This is significant because it means your scraper code doesn’t need to account for these concerns. You write the extraction logic, and the platform handles the unblocking automatically.

• AI-powered self-healing: This is arguably the most operationally relevant feature for running scrapers long-term. When a target site changes its structure, and your scraper breaks, the self-healing tool lets you describe the fix in plain English. The AI analyzes the current code, proposes changes with a diff view, and you approve or reject them. This means that you remain in control of the overall process, as changes aren’t auto-applied unless you explicitly enable auto-approval

• Scheduled runs and data delivery: Scraper Studio includes a scheduling layer, so you can configure periodic runs without external orchestration. Combine this with native delivery to S3, Snowflake, GCS, or Azure, and you have a complete data pipeline that starts from scraping and ends with structured output in your data warehouse, with no spaghetti code required.

• CLI and agent integration: If you prefer working outside the browser UI, scrapers can be built and triggered via the Bright Data CLI, and the AI Agent mode is accessible programmatically, currently compatible with tools like Claude Code, Cursor, and Codex. This means you can integrate Scraper Studio into existing developer workflows without being locked into the web interface.

• Wide pricing offer: The pricing offer may seem like a secondary theme, but it’s not. First and foremost, Bright Data offers you 5k page loads for free every month, backed by their expert support if you need it. The other pricing options are:

  • Pay-as-you-go: $1.5 per 1k page loads, and you pay only for successful requests.

  • Scale: $400/Mo that includes about 400k page loads per month and more.

  • Enterprise: A custom pricing tier for enterprises, based on their specific needs.

Scraper Studio Best Use Cases

If you’re already familiar with Bright Data products and offers, you may be asking why use Scraper Studio. Below is a list of practical examples when this is the right tool to use:

• No APIs or ready-to-use datasets: If you need to scrape a site that isn’t available on Bright Data’s pre-built Datasets Marketplace or on its web scraping APIs, and you want to own the scraper logic yourself but not the infrastructure, well, Scraper Studio is your best choice.

• Recurring pipelines: When you’re building a recurring data pipeline and don’t want to manage cron jobs, deployment infrastructure, or proxy pools separately.

• Quick prototypes: The AI Agent mode can go from a single URL to a working scraper in under 10 minutes. This makes it genuinely useful for validating whether a scraping approach is feasible before committing engineering time on a larger scale.

• You’re in a team with mixed technical levels: The AI Agent is accessible to non-engineers, and the IDE gives developers full control. They both produce the same type of scraper, deployable on the same infrastructure, and as it is interchangeable, your team can manage the same project with different technical levels.

How to Build a Web Scraper With Scraper Studio Using the AI Agent Mode

Time to take the theory into practice. In this section, you’ll learn how to create a web scraper with Scraper Studio using its AI agent mode. The target URL for this tutorial is from the Kappa’s website:

The scraper will retrieve the data from this target page, using the constraints via the prompt description.

Follow the next steps to build your scraper!

Step #1: Create a New AI Web Scraper Project

After creating a new Bright Data account, go to the Scrapers tab and click on Develop a web scraper under New:

You’ll be redirected to a page where you can insert the target URL and the prompt. For this tutorial, the prompt is the following:

Extract the product name, price, available variations / colors, sizes, image URLs, product description, product details.

Below is the visual result:

Note that this is also where you can immediately go for the IDE mode by clicking on Open IDE.

When you’re ready, click on Start scraping with AI, and Scraper Studio will make its magic!

Step #2: Analyze and Approve the Output Schema

The system will automatically produce the output below:

This is the output schema. It’s a structured list of data fields the scraper will extract, along with their data types. Compare this with the data in the target page, and click Approve if the output matches your expectations.

After your approval, Scraper Studio will start creating the web scraper. When the process is done, you’ll receive the following pop-up:

You’ll also get a notification email when the scraper is ready:

All right, your scraper is ready for data retrieval.

Check the TWSC YouTube Channel

Step #3: Manage Preferences and Retrieve The Data

When the scraper is ready, this doesn’t mean the data is scraped. Before scraping the data, you have to manage your preferences.

First of all, go into your scrapers and navigate to the Delivery preferences tab:

Here is where you can choose among:

• Data format for local download, like JSON, NDJSON, CSV, or XLS.

• Third-party connections like Amazon S3 bucket, AWS, Snowflake, and more.

• Get notified via email based on the status of the data delivery.

If you want a scraper that starts recurrently, navigate to the Subscriptions tab and set your schedule:

If you just want to test things around or if you need a one-time scraper, the Initiate manually tab is already sufficient. You just need to click on Start, and the scraper will begin retrieving the data:

Very well! You made it to retrieve data with your first scraper made with Scraper Studio.

Step #4: Check Runs and Go to IDE Mode

To check the status of each scraping run, go to the Runs section. This will display all the information you need on each data retrieval performed by the scraper:

The Actions section allows you to download the data in different formats. Below is the result in JSON:

[
  {
    "product_name": "Blue Men's Trackhouse Moto GP Team Motorcycle T-Shirt with Contrast Inserts",
    "product_subtitle": "AMIRILORE TRACKHOUSE Blue Diva-Grey Antrachite-NEON Yellow",
    "brand": "KAPPA",
    "price": {
      "value": 67,
      "currency": "USD",
      "symbol": "$"
    },
    "sku": "Style: 322J6HW - Color: A02",
    "available_sizes": [
      "XS",
      "S",
      "M",
      "L",
      "XL",
      "2XL",
      "3XL",
      "4XL",
      "5XL",
      "6XL"
    ],
    "image_urls": [
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02_5f5f425b-48d1-4181-af4c-ce863229cbf4.jpg>",
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02_aa19b0fa-ca41-4894-ad74-8d408cc31725.jpg>",
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02.jpg>",
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02_8bd76dc9-0550-48f9-a974-e3c38a4b707e.jpg>"
    ],
    "description": "Product Description GARA. T-shirt for man in polyester fabric. REGULAR FIT. Short sleeve, roundcollar, insert on shoulders in contrast color, sides and shoulders inserts. Omini logo, Team personalizations and sponsors sublimated. TRACKHOUSE.",
    "product_details": "Description Active Jerseys Gender Man Season Spring Composition Main Fabric: Recycled Polyester 100%",
    "input": {
      "url": "<https://www.kappa.com/products/active-jerseys-man-amirilore-trackhouse-blue-diva-grey-antrachite-neon-yellow-322j6hw-trh-a02>"
    }
  }
]

As you can see, the results are exactly how’d you expect.

To change the mode to the full IDE one, navigate to the Code tab:

This is where you can write your custom JavaScript to modify anything you need.

Terrific! You completed your first scraping project with Bright Data’s Scraper Studio.

Need help with your scraping project?

Comparison Table: Scraping via Custom Code Vs Using Scraper Studio

The table below summarizes the differences in scraping the same target using a scraper you coded yourself or using Scraper Studio:

Conclusion

In this article, you discovered what Bright Data’s Scraper Studio is, its features and best use cases, and how easy it is to use it.

As you’ve learned, Scraper Studio is an ideal solution for building a web scraper without any infrastructure overhead, but also without managing selectors manually. Thanks to it, you don’t have to deal with common challenges like infrastructure maintenance, proxy rotation, and anti-bot bypass, but also with DOM changes.

In a matter of a few minutes, you can go from a prompt written in natural language to a production-ready scraper, with automated data delivery to Amazon S3, Snowflake, and many other destinations, and with the possibility to schedule your scrapers and manage DOM changes without manual overhead.

So, let us know in the comments: have you ever tried Scraper Studio? What’s your experience with it?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Retrieval Augmented Generation (RAG) was designed to solve exactly such limitations. By decoupling what the model knows from what it can access at query time, RAG gives LLMs the ability to reason over fresh, domain-specific, and proprietary knowledge without any retraining. In theory, this closes the knowledge gap entirely. In practice, the quality of a RAG system is bound entirely by the quality of its retrieval layer, and this is where most implementations break down.

Indeed, many teams think that the solution is already offered by web search tools that, nowadays, are integrated in all commercial LLMs. This article makes the case for why that is the wrong call and why web scraping is the right retrieval layer that RAG actually requires.

Let’s get into it!

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

What is RAG?

Retrieval Augmented Generation (RAG) is an architectural pattern in natural language processing that augments the inference pipeline of a generative large language model (LLM). It does so by dynamically injecting externally retrieved, contextually relevant information into the prompt at query time. In simple terms, rather than relying exclusively on knowledge encoded in model weights during training, RAG decouples parametric memory (what the model learned) from non-parametric memory (an external, queryable knowledge store), combining both at runtime. This produces models’ outputs that are accurate and grounded.

The canonical RAG pipeline operates in two distinct phases:

1. Retrieval phase: Given a user query q, a retriever component encodes q into a vector embedding database and performs approximate nearest-neighbor (ANN) search against a pre-indexed corpus of document chunks.

2. Generation phase: The retrieved passages are concatenated with the original query and injected into the LLM’s context window as an augmented prompt. The model, then, conditions its output token generation on both its parametric weights and the retrieved evidence, producing a response that is grounded in the supplied documents.

Modern RAG implementations have evolved beyond this baseline. Advanced RAG patterns introduce query rewriting, hypothetical document embeddings (HyDE), re-ranking layers, and recursive or agentic retrieval loops. Modular RAG frameworks further decompose the pipeline into interchangeable components, enabling fine-grained control over each retrieval and generation step.

At the infrastructure level, modern RAG pipelines rely on three core components working in concert:

• A vector database for storing and querying document embeddings at scale.

• An embedding model responsible for encoding both documents and queries into a shared latent semantic space.

• A generative model (the LLM) that synthesizes retrieved evidence into a coherent, task-specific response.

Below is an image that shows the high-level architecture:

Why LLMs Need RAG

During the training phase, an LLM compresses knowledge from its training corpus into billions of parameters. This process is inherently lossy because the model doesn’t store facts verbatim: it learns statistical associations between tokens. As a result of this process:

• Factual recall is probabilistic, not deterministic: The model may confidently produce a plausible-sounding but incorrect answer on a user query, because it’s sampling from a learned distribution rather than retrieving from a verifiable source. The process of producing a plausible-sounding but incorrect response is called hallucination.

• Knowledge is frozen at the training cutoff: Any event, document, regulation, product update, or research finding that postdates the model’s training window is entirely invisible to the model. For enterprise applications where data changes frequently, this is a critical failure mode. This means that if you query a model on something that occurred after its training ended, it will surely hallucinate the response.

• Specific, private, or proprietary knowledge is absent by design: Internal documentation, CRM records, legal contracts, and domain-specific corpora are never part of a pre-training dataset.

RAG was designed precisely to address these constraints that are inherent to LLMs. Below is how RAG maps to concrete business scenarios that enterprises face every day.

Documentation-Aware Support Bots

One of the most immediate and impactful enterprise applications of RAG is the construction of intelligent support bots grounded in a product’s technical documentation. Consider a SaaS company with several pages of technical docs, API references, changelogs, and troubleshooting guides. An LLM cannot reliably answer user questions about that product. What happens is that it either lacks the knowledge entirely or, worse, generates plausible-but-wrong instructions that erode user trust.

With RAG, every user query triggers a retrieval step against the indexed documentation corpus. The bot responds based on what the docs actually say, not on what the model statistically associates with similar-sounding topics.

The same pattern extends naturally to internal knowledge bases like HR policy portals, IT helpdesks, onboarding assistants, and legal FAQ systems, where the source of truth is a controlled, company-owned corpus rather than the public web.

Legal and Regulatory Intelligence on Recent Developments

The legal industry represents one of the clearest demonstrations of RAG’s necessity. Why? Because court rulings, regulatory guidance, legislative amendments, and compliance frameworks are published continuously. Legal research on an LLM is inherently dangerous because its training cutoff means the model is blind to any case law or regulation published after that date. And, in law, a single overlooked precedent can have material consequences.

RAG enables a fundamentally different architecture: legal documents, newly published court opinions, and regulatory updates are scraped, processed, and indexed into the retrieval layer as soon as they are available. When a lawyer or compliance officer queries the system, the retriever fetches the latest relevant documents, and the LLM synthesizes a grounded answer.

Competitive Intelligence and Market Monitoring

Enterprises routinely need to track competitor activity, industry trends, and market signals across news sources, press releases, job postings, and public filings. This information is inherently ephemeral because a competitor’s product launch announcement published yesterday is invisible to any LLM. By building a RAG pipeline that continuously ingests and indexes targeted public web sources, organizations can deploy analysts-in-the-loop systems that answer strategic queries based on fresh, real-world evidence rather than stale parametric knowledge.

Why Web Search Integration Is Not Enough

On the side of the architecture, the retrieval part of RAG systems can be built in different ways. The common ones are two:

• Creating a knowledge base of documents (like a plethora of PDFs and similar).

• Retrieving the needed information from the live web.

The knowledge base of documents is generally something that companies have already built over the years. So, let’s focus on the web retrieval part—which is also the most interesting one for us, scraping professionals.

Today, every commercial LLM has a web search tool integrated into it. So, at first glance, it seems to be the right tool to use for creating RAG systems. After all, if the model can query the live web, doesn’t that solve the knowledge cutoff problem entirely? The answer is: partially, and only for a narrow subset of enterprise needs. This is because the web search integration and RAG are fundamentally different mechanisms that solve different problems. Below are the main differences between the two:

• Web search operates on public web content: First of all, a web search tool cannot reach your internal knowledge base, your proprietary documentation, your CRM history, or your company’s legal contracts. Sure, some companies put their internal knowledge on the web with restricted access, but this encompasses only a few cases. Also, the truth is that no company actually puts all its documentation on a website, even if with restricted access (think of legal contracts, for example).

• Web search is non-deterministic and uncontrolled by design: When an LLM issues a search query, the results it retrieves depend on the search engine’s ranking algorithm, which is commercially influenced and optimized for general web traffic, not for your specific domain. In a RAG system, instead, you control the corpus. In other words, you decide what documents are indexed, how they are chunked, how they are weighted, and when they are refreshed. This level of governance is not optional in regulated industries where compliance is a requirement. Indeed, an enterprise cannot defend an AI-generated legal or financial output that was grounded in whatever Google happened to rank first. Also, no one guarantees that the web search is actually able to read the content on a website that is highly protected. In other words, very often happens that the web search tool can’t overcome the anti-bot systems on a target website, and the model simply hallucinates the response to your query (and you don’t even notice!).

• The retrieval mechanism itself is categorically different: Web search tools rely on keyword-based and PageRank-style relevance signals. RAG retrieval, by contrast, operates in a dense semantic vector space. This means that it finds documents that are conceptually relevant to a query even when they share no exact keywords with it. For domain-specific queries, this distinction in retrieval precision directly impacts the quality of the generated response.

• Web search tools introduce latency, rate limits, and external dependencies: A RAG system backed by an internal vector store is a self-contained, auditable, scalable infrastructure component. A system that depends on a web search tool at every call is coupled to external services, subject to availability failures, and difficult to reproduce or audit

• Web search tools don’t solve the hallucination problem: Web search tools increase the probability that relevant information surfaces in the context window, but if the search query is poorly formed, if the right page isn’t indexed, or if the model misreads the retrieved snippet, hallucination remains entirely possible. RAG, when implemented with rigorous chunking strategies, high-quality embeddings, and re-ranking layers, provides a far more controlled and reliable grounding mechanism.

In short, web search tools make general-purpose LLMs more useful for casual queries. This means that an LLM equipped with a web search tool is useful to respond to basic queries that rely on data recently published on the web. However, they can’t replace RAG systems for any organization that requires precision, privacy, control, and auditability over wide and specific knowledge.

Check the TWSC YouTube Channel

Web Scraping as the Data Acquisition Layer for RAG

The limitations of web search tools presented in the previous paragraph are structural. The solution to overcome them is to use web scraping as the retrieval layer for your RAG pipelines. This is because by using web scraping, you are not querying an external search engine and accepting whatever it returns. You get direct, programmatic access to the content of any publicly accessible web page, on your own schedule, under your own control.

So, let’s see how scraping the web changes the dynamic of retrieving content from the web for RAG purposes.

Full Control Over the Knowledge Corpus

When you build a RAG pipeline fed by a web scraper, you are the one defining the boundaries of your knowledge base. You decide which domains, subdomains, and URL patterns are in scope. You decide how frequently each source is re-crawled, which pages are included, which are filtered out, and which are prioritized during indexing. None of this is possible with web search integration, where the retrieval scope is, by definition, the entire public web as ranked by an opaque external algorithm.

This level of corpus governance is a prerequisite for any enterprise deployment where the knowledge base must be auditable, bounded, and defensible.

Also, using web scraping as the retrieval layer doesn’t mean that you have to code and manage your own scraping infrastructure—which could cause overheads as well as a huge amount of investments. The scraping industry today is mature enough to offer you:

• Pre-scraped datasets whose data can be refreshed based on your needs.

• Scraping infrastructure as-a-service. In other words, you tell the provider the content to scrape, and their professionals do the job for you, providing you with ready-for-analytics data.

Accessing Content That Search Engines Cannot Deliver

Public search engines index a fraction of the publicly accessible web, and they do so on their own timeline and according to their own crawl priorities. Pages that were published recently, that exist on domains with low authority, that are dynamically rendered via JavaScript, or that are structured in ways that resist standard crawling are routinely underrepresented or absent from search engine indexes entirely.

A purpose-built web scraper is not subject to any of these constraints. It can be engineered to handle JavaScript-rendered single-page applications using headless browser automation tools such as Playwright or Puppeteer, capturing the fully hydrated DOM rather than the bare HTML shell that a naive HTTP request would return. It can navigate paginated content, follow internal links recursively to achieve full site coverage, and process non-HTML formats.

Now, you can also think that a plausible solution is to put a target URL into the prompt and ask the LLM to retrieve the content via the web search tool. But due to the nature of the web search tool, this is not a reliable solution, because, as discussed previously:

• The web search tool may not be able to retrieve the content if the domain is highly protected by anti-bots.

• The use of a web search tool is non-deterministic due to how LLMs work. This means that the LLM can respond without actually retrieving the content. Or, in some cases, the LLM can respond, but its response can be a mix of content retrieved from the target URL and other content that the web search tool has retrieved from the web. In other words, the LLM can autonomously decide to retrieve the content, for a given query, from the target URL you passed into the prompt, and from other content that the web search tool retrieves from the web, based on the query.

But for RAG pipelines that need comprehensive, deep coverage of a specific web domain, the right degree of control over the crawling behavior is a fundamental feature, often a non-negotiable one.

Need help with your scraping project?

Deterministic, Reproducible Retrieval

One of the most underappreciated properties of a scraping-fed RAG system is reproducibility. When a user queries a RAG system backed by a web-scraped and version-controlled vector store, the retrieval behavior is deterministic: the same query against the same index returns the same results. This makes the system testable, debuggable, and auditable in ways that a live-search-grounded system categorically is not.

In a search-integrated LLM, the retrieved content is ephemeral. Specifically speaking, the content exists only in the context window of a single inference call. This means that it cannot be logged in a structured way, and may differ entirely if the same query is issued five minutes later because the search ranking shifted. In a scraping-fed RAG system, instead, every document in the index has a known origin, a known ingestion timestamp, and a known embedding. If an output is questioned, you can trace it back to the exact chunk, the exact source page, and the exact crawl that produced it. This traceability is the operational backbone of trustworthy enterprise AI.

Freshness on Your Terms

The knowledge cutoff problem that cripples base LLMs does not disappear with a static RAG corpus. A vector store that was indexed six months ago and never refreshed is, from a knowledge currency standpoint, equivalent to a model with a six-month-old training cutoff. Web scraping solves this by making freshness a configurable, automated pipeline.

A production RAG scraping pipeline typically implements a tiered crawl schedule calibrated to the update frequency of each source: high-volatility sources like breaking news feeds, regulatory update pages, earnings release portals, and product changelog endpoints are re-crawled on a schedule ranging from hourly to daily. Lower-volatility sources may be re-crawled weekly or triggered by change detection signals. When new content is detected, only the affected chunks are re-embedded and re-indexed, making incremental updates computationally efficient even at scale.

This architecture gives enterprises something that neither a base LLM nor a search-integrated model can offer: a knowledge base with a freshness SLA. This is a contractual guarantee that the information grounding the system’s outputs is no older than a defined threshold.

Clean, Structured Text as a First-Class Output

Web scraping, when implemented correctly, does not simply dump raw HTML into the processing pipeline. A production-grade scraping pipeline extracts clean, structured, semantically meaningful text from the web.

This matters enormously for chunking quality, and by extension, for retrieval quality. Recall that the integrity of every chunk is bounded by the integrity of the raw text it was derived from. A scraper that faithfully extracts clean, well-structured prose gives the downstream chunker the structural signals it needs to produce semantically coherent chunks. A scraper that emits raw HTML soup, or that fails to distinguish between primary content and navigational boilerplate, contaminates the entire corpus with noise that no chunking strategy can fully recover from.

Furthermore, this is another point of distinction with web search tools. In another article here at the web scraping club, we already covered the importance of feeding LLMs in markdown, rather than in HTML. Based on this, consider that:

• Every scraping pipeline can be managed to return data in any format you prefer. It can be a structured JSON, markdown, or anything else.

• Web search tools go directly to “read” the HTML of the target web pages. This means that the LLM responds based also on footers, banners, and anything in between. And this can lead the model to confusion as it responds to users’ queries.

Conclusion

RAG is an architectural commitment and, as such, it must be preceded by a deliberate choice to build a system where outputs are grounded in a controlled and continuously refreshed knowledge base. But that commitment is only as strong as the pipeline feeding it.

Web search integration tools are not that pipeline. This is because they are optimized for general-purpose queries, not for the precision, governance, and reproducibility that enterprises demand.

Web scraping, on the other hand, gives you full ownership over what enters the knowledge base, how fresh it is, how it is structured, and how it can be traced back when an output is questioned.

The next step is to start thinking of your scraping infrastructure not as a standalone data collection tool, but as the retrieval backbone of an intelligent system.

So, let us know in the comments: are you using web scraping for RAG purposes?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Instead, the site loads a basic shell and then fetches the live content from an internal API, usually as JSON, which the frontend then renders. E-commerce sites rely on this approach the most, since stock and prices change all the time and nobody wants to redeploy a page for every update. In fact, this is how modern frontends work.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

This actually works in our favor. Internal APIs are much more stable targets than the HTML around them. If the site gets a redesign, all your selectors might break, but the JSON endpoint usually stays put because the frontend still needs it. They’re also easy to spot, since they use regular HTTP in the browser, so you can watch them and see exactly what comes back. The real challenge is finding the right request and figuring out how to reach it past any anti-bot protection.

I always start by hand, in the browser, because that’s the best way to build a mental model of the target and keep your skills sharp. After that, I move to code, since clicking around DevTools for every new site gets tedious, and a small interception script can turn a twenty-minute manual search into a quick automated run. For this example, I’ll use Harrods (https://www.harrods.com/en-it/) as the target. By the end, we’ll be pulling the full product catalog straight from the API the frontend uses.

Doing it by hand first

Open DevTools, go to the Network tab, and filter to Fetch/XHR. That filter drops the images, fonts, scripts, and CSS, and leaves you with the requests the page makes to move data around. On a well behaved site you reload the page, watch a handful of JSON calls show up, and one of them is obviously the product list.

From identifying internal API calls to analyzing structured responses, LLMs are transforming web research. anyIP powers the access layer with residential and mobile IPs built for modern web-data workflows.

Explore anyIP Proxies →

Harrods doesn’t make it that easy, and the reason is worth understanding because it’s common. When you load a category page like women-new-in, the products are already in the HTML. The page is server side rendered, so the first response comes back with the grid baked in and no separate product API call fires. If you only look at the initial load, you’ll conclude there’s no API and reach for HTML parsing.

The API is there. It just doesn’t fire until the page needs new products it doesn’t already have. So you paginate. Click to page two, and now the frontend has to go get sixty products it doesn’t have in memory, so it makes the call. This is the habit that pays off most when you hunt internal APIs. You interact with the page and watch what fires. Paginate, filter, sort, scroll. The data call shows up when the page needs data.

Do that on Harrods with the Network tab open and one request stands out from the analytics noise, a POST to /api/search/1/indexes/*/queries. That path is a giveaway. It’s the Algolia query API, and we’ve pulled product data out of Algolia before (https://substack.thewebscraping.club/p/scraping-algolia-endpoints). Harrods proxies it through its own domain instead of calling Algolia directly, which turns out to matter, but the shape is familiar.

That’s how I do it manually. It works, and honestly, it teaches you more about the target than any tool. But it’s slow, and if you need to check ten sites in an afternoon, it just doesn’t scale. So I automate the search instead.

What Harrods actually is

Before I run any tool, I like to map out the layers, since each one is a possible failure point.

The storefront is a Nuxt application, which we can see from the /_nuxt/builds/meta/*.json calls in the traffic. Nuxt server renders the first page and then hydrates into a single page app, which is exactly why the product grid is in the initial HTML but pagination is a client side fetch. Commerce runs on SCAYLE, visible from the hrd-prd.middleware.api.scayle.cloud and hrd-live.cdn.scayle.cloud hosts. Product search is Algolia, proxied through harrods.com/api/search. Currency and pricing for the Italian storefront are handled by Global-e, which shows up as the gepi.global-e.com calls.

On top of all that, there’s Akamai Bot Manager. This is the layer you have to get past just to load the page, and it decides if you can even use an automated browser. I’m not going to reverse engineer Akamai’s sensor here. That’s another story. Instead, I’ll sidestep it using a stealth browser and a warm-up navigation, then focus on the data. For this walkthrough, the real target is the JSON API layer. Akamai is just the wall I need to climb to get there.

Building the interceptor

The core of the tool is a handler attached to the browser’s response event. For every response, I check the content type, keep the JSON ones, drop the analytics noise, and save the body. Here’s the handler, from intercept.py:

def make_handler(captured, bodies_dir):
    seen = set()

    async def on_response(response):
        try:
            headers = await response.all_headers()
        except Exception:
            return
        ctype = headers.get("content-type", "").lower()
        if not any(m in ctype for m in JSON_MARKERS):
            return
        if is_noise(response.url):
            return

        request = response.request
        base = response.url.split("?")[0]
        key = (request.method, base)
        if key in seen:
            return
        seen.add(key)

        try:
            raw = await response.body()
        except Exception:
            raw = b""
        ...
        print(f"  [json] {request.method} {response.status} {base}  ({len(raw)} B)")

    return on_response

Check the TWSC YouTube Channel

For the content type filter, I use application/json, application/ld+json, text/json, and anything ending with +json. To cut out the noise, I block hosts that are almost always analytics or consent, Google, Facebook, OneTrust, Segment, and similar, so they don’t drown out the real endpoints. I deduplicate by method and path, since hitting a paginated endpoint five times is still just one endpoint. Every response I keep gets its body saved to disk, so I can check the payloads later.

Attaching it is one line, right after I open the page:

page.on("response", make_handler(captured, bodies_dir))

There are two details that make this work on a real site.

First, you need a warm up. If you go straight to a deep URL like women-new-in, Akamai will block you, since you’re asking for an inner page without the cookies a real browser would have collected. So I always load the homepage first, let Akamai set its _abck and bm_sz cookies, wait a bit, and only then go to the category page in the same session.

if warmup:
    print(f"  warmup -> {warmup}")
    await safe_goto(page, warmup, settle_ms * 2)
if await safe_goto(page, url):
    await drive(page, scrolls, settle_ms)

Second, there’s pagination. From the manual check, I know the product API call only happens when the page needs products it doesn’t already have. Just navigating to ?page=2 isn’t enough, since that triggers another server-rendered load. What you need is a client-side click, just like a real user would do. The most reliable way is to trigger the click through the DOM. Playwright’s normal click kept timing out on the pagination control.

async def click_through(page, selectors, clicks, settle_ms):
    for selector in selectors:
        for n in range(clicks):
            count = await page.locator(selector).count()
            if not count:
                print(f"  click skipped: {selector} not in DOM")
                break
            await page.evaluate("() => window.scrollTo(0, document.body.scrollHeight)")
            await page.wait_for_timeout(1000)
            try:
                await page.eval_on_selector(selector, "el => el.click()")
                print(f"  js-click [{n + 1}] {selector}  (matches={count})")
                await page.wait_for_timeout(settle_ms)
            except Exception as exc:
                print(f"  click failed: {selector} -> {type(exc).__name__}")
                break

Running it

Everything here runs from the shared venv, where Playwright (1.55.0), Camoufox (0.4.11), and curl_cffi (0.14.0) were already installed.

First the control. Plain Playwright, headless, straight at the category page:

python intercept.py \
  --url "https://www.harrods.com/en-it/women-new-in" \
  --engine playwright --headless --scrolls 3 --out runs/harrods_vanilla

It doesn’t even get a page back:

playwright._impl._errors.Error: Page.goto: net::ERR_HTTP2_PROTOCOL_ERROR
  at https://www.harrods.com/en-it/women-new-in

That’s Akamai dropping the connection at the HTTP/2 layer, before any HTML is served. It didn’t like the TLS and connection fingerprint of an unpatched automated browser. Worth noticing that this is a different failure than a block page. There’s no “Access Denied” body to read, the socket just closes. The control confirms plain Playwright is a non starter on this target, which is the whole reason the control exists.

Now Camoufox, with the homepage warm up and a click through to page two:

python intercept.py \
  --url "https://www.harrods.com/en-it/women-new-in" \
  --warmup "https://www.harrods.com/en-it/" \
  --click 'a[href="/en-it/women-new-in?page=2"]' --click-times 1 \
  --engine camoufox --scrolls 6 --settle-ms 3000 \
  --out runs/harrods_jsclick

This time the page loads, the title comes back as Designer Women New In | Harrods IT, and the harness prints the JSON calls as they land. Stripping the Akamai sensor beacon and the third party hosts, here’s what Harrods itself serves as JSON:

GET  200 /_i18n/10acb0d1/en-gb/messages.json            (54186 B)
POST 200 /api/rpc/getUser
POST 200 /api/rpc/getBasket                             -> basket
POST 200 /api/rpc/getWishlist                           -> key,items
POST 200 /api/rpc/getHashedString                       -> hash
POST 200 /api/rpc/getExternalIdpRedirect                -> harrods
GET  200 /_nuxt/builds/meta/d3043d4e-...-631f954ba023.json
POST 200 /api/search/1/indexes/*/queries               -> results  (488909 B)

The /api/rpc/* calls are the storefront’s own internal RPC endpoints for things like the basket, wishlist, and user session. These are interesting and worth mapping if you’re modeling the site, but they’re not what I was after. The last line is the one I wanted: a 488 KB POST to the Algolia proxy, keyed on results, and it only showed up after clicking to page two. On the first load, it wasn’t there, just like I saw in the manual check.

Reading the payoff

Here’s what came back in that response:

top keys: ['results']
result[0] keys: ['hits', 'nbHits', 'offset', 'length', 'facets', 'query', 'index', 'queryID', ...]
index: prod_harrods.com | nbHits: 1199
hits on this page: 60

Sixty products in one call, out of 1199 in the category, from the index prod_harrods.com. Each hit carries what you’d actually want:

- brand='Rodo'          name='Satin Dafne Slingback Heels 75'   price={'GBP': 79800, 'EUR': 100000, 'USD': 110000, ...}
- brand='Roksanda'      name='Wool Scarf-Tie Isabel Blazer'     price={'GBP': 139500, ...}
- brand='Alessandra Rich' name='Silk Polka-Dot Bow Dress'       price={'GBP': 160000, ...}

You get product id, reference key, name, brand, gender, URL, images, category hierarchy, and price as a map of currencies in minor units. For example, GBP: 79800 means £798.00. This is exactly the same data you see in the product grid, but now you can store it directly without having to parse the HTML.

The reusable part is the request itself. Here’s the POST body the frontend sent:

{"requests":[{"indexName":"prod_harrods.com","query":"",
  "params":"facets=%5B%22*%22%5D&clickAnalytics=true&ruleContexts=%5B%22EUR%22%5D&maxValuesPerFacet=1000&facetFilters=%5B%5B%22categoryPageIdentifier%3AWomen%20New%20In%22%5D%5D&offset=60&length=60"}]}

If you decode the params, it’s all clear. The facetFilters with categoryPageIdentifier:Women New In limits the query to this category. offset=60&length=60 handles pagination, so page two starts at offset sixty. Change the offset and you can walk through the whole category, sixty products at a time.

One thing is clearly missing here. There’s no x-algolia-application-id or x-algolia-api-key, which a direct Algolia call always carries. That’s because we’re not calling Algolia, we’re calling harrods.com/api/search, and the proxy injects the credentials server side. We don’t need to find or rotate an Algolia key, we just POST to the Harrods path with the cookies our browser already earned.

Replaying it to page the whole catalog

Read more

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

The rollercoaster inside the proxy industry

Residential proxy networks are facing a hard time. After the IPIDEA case, the FBI targeted NetNut, one of the biggest names in the industry, and seized their domain netnut.io, proxyjet.io (subsidiary company), and divinetworks.com (partner network for IP sourcing). The suspicion is that their residential proxy network was used for fraudulent operations.

I’ve worked closely with NetNut, and I don’t think they’re directly involved in anything: it’s a public company, there’s no need to put their network at a fraudster’s disposal. But some pieces of the KYC procedures and auditing were probably missing, given that they also had a network of resellers which are harder to control.

But the hard news is not ending here: following the Spur research on Smart TVs we shared last month, LG decided to ban the residential proxy option from their apps on the webOS platform.

This crusade against residential proxy networks probably will lead to major shifts in the next months (ready to be contradicted).

• Less supply, higher demand: higher prices and/or worse quality. At least for residential proxies, the price run to the bottom will stop. With demand growing and fueled by AI needs and less supply of IPs, more requests will transit through the same addresses, degrading quality. More providers will try to avoid this by charging more for premium-quality addresses (read: addresses with fewer requests on them).

• No more easy access to proxies: some companies have already restricted the usage of residential networks to customers who have already completed the KYC process.

  

  In this example, Bright Data is requiring KYC for using residential proxies. Until a few months ago, it was required only when targeting specific websites.

An interesting point of view about this situation can be read in an article from (Massive founder) on this Substack.

Jason's Substack

The Web Needs a Front Door

The web’s original breakthrough was radically low-friction access. CERN released its core technology royalty-free, allowing anyone with an internet connection to publish information, link to others, and build without asking a central platform for permission…

Read more

25 days ago · 5 likes · Jason

A rising tide lifts all boats

Luckily, we don’t have just bad news from the scraping industry.

After the Nimble February announcement of a Series B funding round of 47M USD, Oxylabs received a 130M USD investment from private equity firm Warburg Pincus.

The need for data by AI, both for training and in real time, is fueling the industry revenues.
But it’s not just a matter of money; it’s about being recognized as a crucial technology layer for every business that needs web data.

Another piece of good news comes from the legal side: Federal Court Grants SerpApi’s Motion to Dismiss Google’s DMCA Lawsuit. This is a major win for the web scraping industry, since “the court rejected Google’s attempts to expand the DMCA to assert control over access to public pages”.

The Napster era for content

I’m not saying anything new when I state that AI overviews and LLMs’ answers are crunching visits to websites (and, in some cases, hammering them with millions of requests for scraping them).
Cloudflare recently announced that, by default, “for new customers and for new sites for existing customers, on September 15, 2026, the defaults will be set to allow for search but block training and agent use for pages with ads”.

The idea, which seems good on paper, is to make AI crawlers pay for data to compensate the publisher instead of scraping it. They already announced the Pay-per-crawl protocol some time ago, and this choice goes in the same direction.

We did the math some time ago, and it doesn’t make any sense. Those fees should be low enough to make it more convenient to pay instead of bypassing Cloudflare. This means that 99% of website owners won’t even notice the payment on their bank account.

I believe we’re entering the Napster era for online content. Premium content that we don’t want to be fully cannibalized by AI will be under a paywall. There will always be someone who pays for it and then shares it for free with others; it’s unavoidable, just like on Napster in the 2000s. Let’s call this unwanted marketing expenses.

Read and share more of this

Want to flag and discover more articles like the ones above? Scraping News (still in beta) is where the community surfaces them. Sign up and submit what you find worth reading.

Want a concept map of everything covered here, cross-referenced with our own work and other sources worth your time?

The Web Scraping Club Wiki lives on GitHub, and as an interactive site. It is plain Markdown, so you can also clone it into your own Obsidian vault and read it locally.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

An Introduction to SearchApi

Before exploring the new MCP offering, let me first introduce you to the world of SearchApi!

What Is SearchApi?

SearchApi is an online service that provides multiple scraping APIs. It allows you to get data from a wide range of websites, with a strong focus on major search engines.

SearchAPI endpoints provide programmatically access to real-time data from Google, Bing, and YouTube, as well as platforms like Amazon, Airbnb, Zillow, TikTok, and many others. It handles the underlying infrastructure for you, while also taking care of proxy rotation and bot protections.

Main APIs

Below is a table of some of the most relevant SearchApi endpoints:

To discover all endpoints, visit the official documentation. On each API page, you can also access the OpenAPI specification, as well as download the Postman collection.

Pricing Plans

SearchApi follows a pay-per-success pricing model built around monthly subscription plans. These range from developer-level access ($40/mo) to high-throughput enterprise tiers ($5,000/mo), each including monthly search quotas and 99.9% SLA guarantees.

Pricing decreases with scale, starting at around $4 per 1,000 searches and going down to about $1 per 1,000 at higher volumes. A free trial is also available, offering 100 free requests before committing to a paid plan.

Introduction to the SearchApi Web Search MCP Server

As a web scraping enthusiast, chances are you’re already familiar with scraping APIs or SearchApi itself. So, since the company has recently launched a new MCP server product (called Web Search MCP), let me focus on that!

How It Works

Web Search MCP enables AI models to connect to SearchApi’s endpoints via a unified interface.

It’s a remote-only MCP server that operates over Streamable HTTP. That means you can’t install it locally. Instead, it runs as a hosted service that your AI client connects via an MCP URL. The server also supports OAuth 2.0, enabling secure authentication.

Thanks to the OAuth support, you can connect SearchApi to your AI tools through a simple “Connect” flow. This way, the AI client can act on the user’s behalf without ever handling or storing the user’s API key or password. Access can be revoked at any time from either the SearchApi dashboard or the connected AI platform, giving you full control over permissions.

Your AI assistant sends requests to the MCP server, which processes them, executes the required actions, and returns structured results in real time. This setup removes the need to manage local infrastructure or dependencies while keeping integration straightforward.

SearchApi’s Web Search MCP server can be connected to any tool that supports remote MCP servers via Streamable HTTP, including Claude Desktop, Claude Code, Cursor, and many others. It’s also available as an official Claude connector.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Tools

The Web Search MCP server exposes over 80 tools. Some of the most useful ones include:

Discover all other tools in the official docs.

Pricing and Limitations

Pricing follows the same pay-per-success model as the regular SearchApi APIs, with each MCP tool invocation counting as a single API request. Web Search MCP supports up to 10 MCP integrations per account. Also, standard SearchApi rate limits apply.

How to Use the SearchApi Web Search MCP Server in a Real-World Scenario

To showcase the SearchApi Web Search MCP Server, I’ll configure it in Claude Code (though any other compatible MCP tool will work as well) and use it in a sentiment analysis workflow. The goal is to retrieve the latest reviews for a restaurant and let the AI analyze them.

Follow the instructions below!

Prerequisites

To follow along with this section, make sure you have:

• A SearchApi account (a free trial account is enough).

• Claude Code installed and configured locally.

• A basic understanding of how MCP works.

Step #1: Create Your MCP Integration

Start by logging in to your SearchApi account. From the dashboard, click “MCP” under the “INTEGRATIONS” section:

Next, select “Your MCPs” and click “New Integration”:

You’ll be taken to the page for creating a new SearchApi Web Search MCP integration. Start by giving your integration a name (e.g., “Review Bundle”).

Next, either select one of the recommended tool bundles or customize your own. To do so, scroll down, search for “review,” and add all review-related tools:

• Airbnb Property Reviews

• Facebook Business Page Reviews

• Google Maps Reviews

• Tripadvisor Reviews

• Walmart Reviews

Now, most review tools require a page URL, place ID, or another entity identifier. To retrieve those automatically, you’ll also need some search tools.

For that reason, search for “Google” and add:

• Google Maps Search

• Google Maps Reviews

The “Google Search Light” tool is fundamental. From the returned search results, AI finds the relevant page URL (e.g., the Tripadvisor page for a specific restaurant), which typically already contains the required identifier or can be used to derive it. This ID or URL can then be passed directly to the corresponding review tools to fetch detailed reviews and structured data.

Once you’ve selected all the required tools, click “Create Integration” to generate your SearchApi MCP integration. You’re now ready to connect it to your AI client. Cool!

Check the TWSC YouTube Channel

Step #2: Configure Your SearchApi MCP URL in Claude Code

After creating the MCP integration, you’ll be redirected to your SearchApi MCP page. Here, you can find your MCP URL along with setup instructions for Claude Code, Claude Desktop, and Cursor.

Copy the MCP URL, which follows this format:

https://www.searchapi.io/mcp?token=<YOUR_SEARCHAPI_MCP_TOKEN>

<YOUR_SEARCHAPI_MCP_TOKEN> is a unique token generated by SearchApi. It identifies your specific MCP integration and is used for authentication.

Note: For enterprise integrations and agent platforms that require delegated access, OAuth authentication is the recommended approach.

Next, add the MCP server called review_bundle to Claude Code using this command:

claude mcp add review_bundle "<YOUR_SEARCH_API_URL>" --transport http

Make sure to replace the <YOUR_SEARCH_API_URL> placeholder with the URL you copied from the dashboard. Amazing!

Step #3: Verify That the MCP Integration Works

Begin by launching Claude Code:

claude

Then run the command below:

/mcp

This will list all available MCP servers, including review_bundle:

Select the review_bundle server, then choose “View tools”. You’ll see all 7 configured tools:

This confirms that your Claude Code environment has successfully connected to the remote SearchApi Web Search MCP tools. Perfect!

Step #4: Test the Workflow

To verify that the MCP integration works for review scraping, try a sentiment analysis prompt like this:

Get the latest reviews for Sottocasa Pizzeria Harlem in New York from Google Maps and Tripadvisor. Then, generate a sentiment analysis report in Markdown that includes quotes from the retrieved reviews, and save it as reviews.md.

Execute the prompt, and you should see something like this:

This is what Claude Code did:

1. Understood that it first needed to retrieve place IDs.

2. Used google_maps_search and google_light_search to find the Google Maps ID and Tripadvisor ID for the restaurant, respectively.

3. Called tripadvisor_reviews to fetch the latest 15 reviews using the Tripadvisor ID, and google_maps_reviews to fetch 15 recent reviews using the Google Maps ID.

4. Aggregated the retrieved reviews in JSON format and generated a structured Markdown sentiment analysis report with supporting quotes.

5. Exported the final output to reviews.md.

Important: Claude’s built-in Fetch and Web Search tools may not reliably achieve this goal due to anti-bot protections on platforms like Google and Tripadvisor.

The final result is a reviews.md file containing a sentiment analysis based on the latest 15 Google Maps reviews and 15 Tripadvisor reviews for the selected restaurant:

Notice how the report includes direct quotes from relevant reviews and real-world insights extracted from them. Mission complete!

[Extra] Analyze the Output of the Web Search MCP Tools

Expand the MCP tool calls made by Claude Code, and you’ll see:

Notice the use of Google Maps Search to retrieve the Google Maps ID of the target restaurant.

Similarly, expand the Google Search Light tool call to inspect the JSON SERP returned by SearchApi for the “Sottocasa Pizzeria Harlem Tripadvisor” query in the US:

Next, focus on the Google Maps review scraping step:

Note how this returns results in a structured array field (reviews) containing all relevant review data. The same pattern applies to the Tripadvisor review tools:

This demonstrates how the SearchApi Web Search MCP enables Claude Code to complete sentiment analysis tasks by providing real-time scraped review data from multiple platforms, returned in a structured format that is easy to process and analyze.

Need help with your scraping project?

Other Supported Use Cases

Beyond review scraping and sentiment analysis, SearchApi’s Web Search MCP unlocks a wide range of use cases. These include:

• Travel planning and price optimization: Search flights, hotels, and vacation rentals while comparing prices and availability across dates. AI agents can recommend optimal travel windows and cost-efficient itineraries.

• Market and ad intelligence monitoring: Analyze ads across Meta, Google, TikTok, and LinkedIn to track competitor messaging, creative strategies, and campaign activity in real time.

• E-commerce product research and comparison: Retrieve product listings, pricing, and reviews across Amazon, Walmart, eBay, Best Buy, and Google Shopping to build product comparison or recommendation systems.

• Local business discovery and mapping agents: Use Google Maps and Apple Maps to find places, extract reviews, generate directions, and build location-aware assistants for restaurants, services, or POIs.

• Social media and brand intelligence tracking: Fetch public profiles from Instagram, TikTok, and Facebook to monitor influencers, competitors, or brand presence and engagement metrics.

• News and trend monitoring agents: Search Google News to track breaking news, industry updates, and topic trends for real-time intelligence dashboards or alerting systems.

• Educational and research assistants: Query Google Scholar to retrieve academic papers, legal opinions, and citations for research summarization and knowledge extraction.

Conclusion

I’ve personally tested MCP servers from various providers in the scraping industry, and the experience with SearchApi’s Web Search MCP has definitely been smooth.

I didn’t encounter any significant issues during setup, and the integration worked immediately. Across the MCP tools I tested, none of the requests failed, which is a strong indicator of reliability and overall quality.

I’d therefore recommend it as a solid solution for giving AI agents rich search capabilities across multiple platforms, especially for companies and developers with some budget (given the lack of a free plan).

I hope you find this post useful. If you have any questions, feel free to reach out in the comments below. Until next time!

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

I wanted a number I could actually trust, not just another marketing claim, and started to write this piece. Then I saw the great work on the Tyler Richards website (richards.foo/tools/stealth-benchmark) and took some inspiration from it. My idea was to take the open source Python automation stack, point each tool at the same signals a bot detector checks in JavaScript, and score them all the same way. In the past, I have written about anti-detect browsers and fingerprint test pages, but those were about commercial GUI browsers. This time, I focused on the libraries you actually pip install and use in code.

The whole test harness is in our GitHub repository reserved for paying users, inside the folder 111.STEALTH_BENCHMARK. You can run it yourself and get your own results.

The tools I put on the bench

I grouped the fifteen tools into families, so the scores are easier to compare.

My two baselines are plain Playwright and plain Selenium, with no stealth and no patches. These are the control group, and they are supposed to get caught. If a “stealth” tool scores the same as vanilla Playwright, that tells you a lot.

The Playwright stealth patches are patchright, rebrowser-playwright, and undetected-playwright. These keep the Playwright API and try to remove specific automation leaks underneath it.

The CDP-native group is pydoll, nodriver, and zendriver. They drop WebDriver entirely and talk to Chrome over the DevTools Protocol. nodriver is the official successor to undetected-chromedriver, written by the same author, and zendriver is a maintained fork of nodriver.

Undetectable browsers are getting impressively capable, but even the stealthiest one stalls when its connection lets it down. anyIP’s residential and mobile proxies pair real-user IPs with 99.9% uptime - so the network is never the weak link

Build with anyIP →

Then the browsers that rebuild the fingerprint instead of patching flags on top of a normal Chrome. camoufox and invisible_playwright are patched Firefox builds that rewrite the fingerprint at the C++ level. CloakBrowser does the same to Chromium, shipping its own patched binary with a drop-in Playwright API. RayoBrowse is a self-hosted stealth Chromium from Rayobyte that runs in a Docker container and serves a fingerprint picked from a database of real devices. Obscura is the odd one out, a headless browser engine written in Rust that embeds V8 and ships its own browser rather than driving Chrome at all. And scrapling through its StealthyFetcher wraps a stealth Chromium, while undetected-chromedriver is patched Selenium.

What the detector actually sees

A bot detector is not just one thing, so before testing, you have to decide which layer you are actually targeting. In practice, a real detector looks at four main areas: the JavaScript fingerprint surface, the transport (TLS and HTTP/2), request timing, and the proxy exit. For this benchmark, I focused only on the JavaScript fingerprint surface and ignored the rest on purpose.

This is not about being lazy. Scoping it this way is what keeps the results honest. Transport fingerprinting mostly depends on the browser’s own network stack, which is the same whether you use Chromium or Firefox, no matter which Python library you use. Timing and behavior are a different topic, more about mouse movement and dwell time. Proxy reputation is about your IP, not your tool. So the real question I can answer by swapping libraries is simple: when a detector’s JavaScript runs inside the page, what does it actually find?

The detector looks for a fixed list of tells. Is navigator.webdriver true. Does the user agent say HeadlessChrome. Is the WebGL renderer a software rasterizer like SwiftShader instead of a real GPU. Are there leftover automation globals on window, like the old chromedriver cdc_ variables or a framework’s init scripts. Does the chrome object have the right properties. Do the user agent, navigator.platform, and the Client Hints all agree on the OS. The knowledge base on richards.foo lists hundreds of these, but I pulled the highest severity ones into my own probe.

How I set up the test

At the core, I use a single HTML page called probe.html, served from a small local server. Each tool loads this page, the page reads its own fingerprint in JavaScript, and writes the result back into the DOM as base64 JSON. I use base64 so I do not run into any HTML-escaping issues when I pull the rendered page from each driver.

To keep the harness manageable, I wrap every tool, no matter how different its API, in a single method:

def render(url, wait_selector=None, timeout=30000, settle_ms=6000, screenshot_path=None) -> str:
    # launch in the tool's stealth config, goto(url), wait, return final HTML, close

The new anti-bot solution - Toughest walls, lowest price. Claim your free 10 000 requests with coupon code WSCLUB

Claim your credits

Whether it is Playwright sync, Selenium’s execute_script, pydoll and nodriver’s async CDP calls, scrapling’s StealthyFetcher.fetch, CloakBrowser’s launch(), RayoBrowse over a Docker CDP socket, or Obscura over its Rust CDP server, they all get wrapped by render(url) -> html. I use the same method for the probe and the two oracle pages, so I never have to special-case a driver. Here is the full vanilla Playwright adapter, and it does not get simpler than this:

 from playwright.sync_api import sync_playwright

def render(url, wait_selector=None, timeout=30000, settle_ms=6000, screenshot_path=None):
    with sync_playwright() as p:
        browser = p.chromium.launch(channel="chrome", headless=True)
        page = browser.new_page()
        try:
            page.goto(url, wait_until="load", timeout=timeout)
            if wait_selector:
                page.wait_for_selector(wait_selector, timeout=timeout)
            else:
                page.wait_for_timeout(settle_ms)
            if screenshot_path:
                page.screenshot(path=screenshot_path, full_page=True)
            return page.content()
        finally:
            browser.close()

Check the TWSC YouTube Channel

For every Chrome-based tool, I used channel="chrome" to drive the real installed Chrome, not Playwright’s bundled Chromium. This choice is important. Playwright’s bundled Chromium in headless mode reports a software WebGL renderer (SwiftShader), which is just a headless artifact, not something the tool did. When I point the same tools at real Chrome, the WebGL renderer shows the actual Apple M2 GPU. By using real Chrome everywhere, the tests I measure reflect the tool’s stealth, not which Chromium binary happened to ship. The engine-level browsers like Camoufox, CloakBrowser, and Obscura ship their own binary, so they use that instead.

I ran everything in headless mode on purpose, since that is both harder and more common for scraping. Headed mode hides some of these tells automatically, and I mention that where it is relevant.

I do not just trust my own score, so I use two outside referees. My probe gives a 0 to 100 score by subtracting a weight for each tell the tool leaks. Then I check two open bot-detection pages for an independent verdict: BrowserScan and deviceandbrowserinfo. deviceandbrowserinfo is more useful because it returns a JSON breakdown, an isBot flag, and about twenty individual detections like hasWebdriverTrue, isHeadlessChrome, and isAutomatedWithCDP. I tuned the weights in my probe so the ordering matches those verdicts, and that is the calibration step. The runner also saves a screenshot of the deviceandbrowserinfo verdict for every tool, so the human or bot call is on record, not just my parser’s output.

Running it, and what came back

Fifteen tools, three targets each, and here is the table the reporter produced.

My probe score and the deviceandbrowserinfo verdict matched on 12 out of 13 tools I could get a verdict for, so that is a 92 percent match. I saved the heatmap of every tell by every tool at 111.STEALTH_BENCHMARK/results/heatmap.png, and you can see the tiers at a glance.

Four tools passed the strict referee: camoufox, CloakBrowser, RayoBrowse, and scrapling. Three of them get there the same way, by rebuilding the browser itself instead of patching automation flags onto a normal Chrome, so there is no HeadlessChrome in the user agent and nothing that reads as CDP automation. scrapling is the fourth, and it passes for a reason worth its own section below, because it took a version bump to get there. Everything that still leaves navigator.webdriver true sits at the bottom of the table, and everything in the middle is one user agent string away from a clean pass.

Read more

Then you get access to a residential proxy pool, and the scraper works.

If you’ve been there, you probably moved on without asking why. The adage is simple: if it works, don’t touch it!

But that “why” matters more than you think because residential proxies aren’t a kind of black magic. They’re a specific technical architecture on top of which your scrapers work. And as with any architectural choice, you have to make decisions on purpose and based on specific parameters.

In this article, you’ll get a complete technical picture of residential proxies, starting from how IP addresses are classified, how the proxy infrastructure routes your traffic, what residential proxies are, and their common types. In the end, you’ll also discover the ethics behind residential IP sourcing and how to choose ethical providers with a practical decision framework.

Let’s get into it!

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

What Is an IP Address?

An IP address (Internet Protocol address) is a numerical label assigned to every device connected to a network that uses the Internet Protocol for communication. On the practical side, the IP protocol serves two core functions:

• Host or network interface identification.

• Location addressing.

The current dominant versions are IPv4 and IPv6:

• IPv4 addresses are 32-bit integers, expressed in dotted-decimal notation as four octets ranging from 0 to 255 (for example, 203.0.113.47), yielding a theoretical address space of ~4.3 billion unique addresses.

• Due to exhaustion of this space, IPv6 was introduced, using 128-bit addresses expressed in colon-separated hexadecimal groups (for instance, 2001:0db8:85a3:0000:0000:8a2e:0370:7334), offering a space of ~3.4 × 10³⁸ addresses.

IP addresses aren’t handed out randomly by providers: there’s a strict hierarchy that they must respect. Specifically, a global authority called IANA divides the total pool of IP addresses into large blocks and distributes them to regional bodies known as RIRs (Regional Internet Registries). These RIRs then assign smaller blocks to ISPs (your home internet provider) and cloud providers (like AWS or Google Cloud), who finally assign individual IPs to end users and servers.

Why does this matter for scraping? Because this entire chain is publicly documented, and anyone can look up who owns a given IP address using tools like WHOIS (a directory of IP ownership records). This means a target website can instantly tell whether your request is coming from a residential IP, a datacenter IP, or a known proxy or VPN provider.

This is precisely why residential proxies are so valuable in web scraping: their IPs are indistinguishable from those of regular home users, making them much harder for anti-bot systems to flag.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

What Is a Proxy Server?

A proxy server is an intermediary host that sits between your client and the target web server, forwarding requests and responses on your behalf:

Without a proxy, the TCP connection to the target server is established directly from your machine. With a proxy, your client opens a connection to the proxy, which opens a separate connection to the target, and the target sees the proxy’s IP address as the request origin. This way, your real IP never appears in the target’s access logs or network layer.

This IP substitution is the fundamental value proposition of a proxy in a scraping context. Every request your scraper sends carries a source IP and, as covered above, target servers actively use that IP to make trust decisions. By routing through a proxy, you externalize your network identity. This way, the anti-bots on target servers evaluate the reputation, geolocation, and classification of the proxy’s IP, not yours.

At the network level, a proxy is simply a process listening on a TCP port, capable of accepting inbound connections and initiating outbound ones. It maintains two separate TCP connections for each proxied request: one with your client and one with the target. These connections are independent: they can use different protocols, different keep-alive settings, and the proxy can pool or reuse the outbound connection across multiple client requests.

The proxy may also perform additional functions depending on its implementation. For example, it can manage authentication, logging, IP rotation (cycling through a pool of IPs across requests), and header manipulation.

Proxy Protocols: HTTP/HTTPS vs. SOCKS5

When routing traffic through a proxy, the protocol determines how the proxy handles your traffic at the application and transport layers. This has real implications for scraping purposes. The main protocols are the following:

• HTTP/HTTPS: Proxies that use these protocols operate at the application layer (OSI Layer 7). When your scraping client sends a request through an HTTP proxy, it constructs a full HTTP request and sends it to the proxy server, which forwards it to the target on your behalf. For plain HTTP traffic, the proxy reads the request in full. For HTTPS traffic over an HTTP proxy, the client issues a CONNECT method request, instructing the proxy to establish a raw TCP tunnel to the destination. Once the tunnel is open, the TLS handshake occurs directly between your client and the target server. This means that the proxy becomes a blind pipe and cannot decrypt the payload. This is critical to understand: an HTTP proxy handling HTTPS via CONNECT does not perform SSL termination (unless it’s explicitly a man-in-the-middle proxy with a custom CA cert injected).

• SOCKS5: Is a lower-level protocol operating at the transport layer (OSI Layer 5). Unlike HTTP proxies, SOCKS5 is protocol-agnostic. This means that it simply relays raw TCP (and optionally UDP) streams without any awareness of the application-layer protocol being tunneled. After a brief handshake, the proxy establishes a TCP connection to the destination and shuttles bytes in both directions without inspection. This makes SOCKS5 suitable for any TCP-based protocol, not just web traffic.

One operationally significant detail to consider is DNS resolution. With SOCKS5, the domain name is forwarded to the proxy for resolution, meaning DNS queries originate from the proxy’s network, not yours. This prevents DNS leaks, which can expose your real network location even when your traffic routes through a proxy. HTTP proxies also resolve DNS on the proxy side for CONNECT-tunneled requests, but behavior can vary by implementation.

Now, here’s a question for you: for scraping purposes, which protocol is the best? Well, for most scraping use cases, either protocol is functionally equivalent from the target server’s perspective, since both can tunnel TLS. SOCKS5 is generally preferred for its flexibility and slightly lower overhead, but HTTP proxies are often easier to drop into existing HTTP client configurations without additional dependencies.

Tired of stitching together proxies, browsers, and parsers just to scrape one page? Gaffa handles the whole stack for you, with the control to fine-tune every step when you need it.

Start With Gaffa

What Are Residential Proxies?

A residential proxy is a proxy server whose outbound IP address is assigned by a consumer Internet Service Provider (ISP) to a real, physical household device like a home router, laptop, smartphone, smart TV, or similar. The word “residential” refers directly to the ISP classification of the IP: these are addresses allocated to residential customers, and they appear as such in public IP intelligence databases.

How Residential Proxies Work

The operational model of a residential proxy network involves three distinct parties: the proxy provider (the company offering proxy servers on the market), the exit node (the residential device), and the scraping client (you).

The exit node is a real consumer device whose owner has installed software that, in exchange for some compensation, allows the provider to route third-party traffic through their internet connection. This software runs as a background process and registers the device with the provider’s orchestration infrastructure. The device’s residential IP is thus made available in the provider’s pool.

The provider’s infrastructure is the backbone that ties everything together. It consists of:

• A gateway layer: A set of proxy endpoints that your scraping client connects to. These endpoints are hosted on standard data center infrastructure and serve as the entry point into the residential network.

• An orchestration/routing layer: The internal system responsible for selecting which residential exit node to use for a given outbound request, based on criteria like target geolocation, session persistence requirements, and node availability.

• The exit node pool: The fleet of residential devices registered with the provider, each maintaining a persistent connection back to the orchestration layer.

Thanks to this infrastructure, the scraping requests flow like that:

1. Your scraping client connects to the provider’s gateway endpoint and authenticates.

2. The gateway receives your request and passes it to the orchestration layer, which selects an available residential exit node from the pool based on your session’s parameters.

3. The orchestration layer forwards the request to the selected exit node over the persistent internal tunnel.

4. The exit node’s software initiates the actual outbound TCP connection to the target server from the residential device’s IP.

5. The target server receives a request from what appears to be a residential IP, processes it, and returns the response.

6. The response travels back through the same path in reverse: exit node → orchestration layer → gateway → your client.

This is why, from the target server’s perspective, the only visible connection is the one from the residential exit node. Your client’s IP, the provider’s gateway IP, and the internal routing infrastructure are entirely invisible.

Check the TWSC YouTube Channel

Types of Residential Proxies

Residential proxies are segmented into distinct types based on two primary axes: IP persistence (how long you hold the same IP across requests) and targeting granularity (how precisely you can select the exit node’s location).

This section reports and describes the common residential proxy categories that are based on those two axes.

Rotating Residential Proxies

This is the most common type. With rotating proxies, the provider’s orchestration layer assigns a new exit node—and, therefore, a new IP—for each outbound request, or at a configurable interval. From your client’s perspective, you connect to the same gateway endpoint every time, but each request exits from a different residential IP. This is the default behavior on most residential proxy platforms and is the correct choice for large-scale scraping where you need to distribute requests across a wide IP footprint to avoid per-IP rate limits. Belos is a schema that shows, at a high level, how this process works:

The rotation can be triggered in two ways, depending on the provider:

• Per-request rotation: A new IP is selected for every single HTTP request.

• Timed rotation: The same IP is held for a configurable duration before rotating.

In the context of web scraping, the primary tradeoff is that rotating proxies make stateful interactions difficult. If a target website tracks session state via server-side session IDs or cookies tied to an IP, switching IPs mid-session will break the session. So, for purely stateless scraping cases like fetching product pages, search results, and public data, rotation is ideal.

Sticky Residential Proxies

Sticky proxies allow you to hold the same residential IP for an extended, configurable period. This typically ranges from a few minutes up to 30 minutes, depending on the provider, and is bounded by the exit node’s availability online. The session is maintained by passing a session identifier in your proxy credentials, which instructs the orchestration layer to pin your requests to the same exit node for the duration of the session.

This type is essential for scraping workflows that require statefulness, like multi-step form submissions, login-based scraping, checkout flows, or any target that correlates sequential requests by IP and invalidates sessions on IP change. The risk is that if the exit node goes offline mid-session (the residential device disconnects from the network, which happens regularly on consumer connections), the session is broken, and the provider will typically reassign you to a new IP. This requires your scraper to handle session recovery; otherwise, your pipelines will break.

Static Residential Proxies

Static residential proxies (often known as ISP proxies) occupy a middle ground. These IPs are registered to an ISP (giving them a residential-looking ASN and WHOIS classification) but are hosted on stable data center infrastructure, rather than actual consumer devices. The provider procures these IPs directly from ISPs and hosts them on servers they control.

The key advantage over standard rotating residential proxies is stability and speed. Since the IP is hosted on a server rather than a consumer device, it doesn’t go offline unpredictably, latency is lower, and bandwidth is not constrained by a household connection. You get a fixed, persistent residential-classified IP that behaves like a data center proxy in terms of reliability.

The tradeoffs are:

• Cost: These are among the most expensive proxy types.

• Increasing detectability: As anti-bot vendors are building databases of known static residential/ISP proxy IP ranges, this kind of proxy works as a cat-and-mouse race to escape anti-bots.

Mobile Proxies

Mobile proxies are a specialized subtype where the exit node is a mobile device connected via a cellular network, and the IP belongs to a mobile carrier’s ASN. These are distinct from standard residential proxies because mobile IPs carry a specific characteristic: mobile carriers use large-scale carrier-grade NAT (CGNAT), meaning a single public IP is shared among potentially thousands of concurrent mobile users.

This has a profound implication for anti-bot systems: blocking a mobile carrier IP risks blocking thousands of legitimate users simultaneously, making mobile IPs extremely difficult for targets to block without significant collateral damage. As a result, mobile proxies are the hardest proxy type to block and command the highest price per GB in the market. They are most valuable for targets with aggressive anti-bot systems, typically heavily protected e-commerce sites, social platforms, and ad verification use cases.

The tradeoff is latency and bandwidth variability. The exit node is on a cellular connection, which is subject to signal quality, network congestion, and higher round-trip times compared to a wired residential or data center connection.

Residential Proxies Types: A Summary Comparison

The table below summarizes the subtypes of residential proxies, each with its own specific characteristics:

Need help with your scraping project?

How Residential Proxies Are Sourced and Why It Matters

The technical sophistication of residential proxy networks is surely interesting for techies. But what’s also interesting is where the exit nodes actually come from. This is because understanding the sourcing model is not just an ethical exercise. It has direct operational implications for scraping professionals because the legality and reputational standing of the proxies you use can affect your liability exposure and the long-term reliability of your infrastructure. It also has implications for companies working in highly regulated markets, where specific certifications matter more than anything else. We’ve seen recently how bad things can go from one day to another if your proxy provider is not fully compliant on both IP sourcing side and customer onboarding.

As introduced previously, providers build their residential IP pools by recruiting real consumer devices to act as exit nodes. The mechanisms for doing this fall into a spectrum that is described in the subparagraphs below.

SDK Monetization

The most widespread IP sourcing method is SDK embedding. In this model, a residential proxy provider operates a B2B program where mobile app developers and software vendors integrate the provider’s SDK into their applications. In exchange, the developer receives a revenue share or a flat fee per active device. When an end user installs the app, the SDK runs in the background and enrolls the device’s internet connection into the proxy pool.

The provider’s SDK typically does the following:

• Establishes a persistent outbound connection to the provider’s orchestration infrastructure.

• Monitors device state to only proxy traffic when the device is charging and connected to Wi-Fi (to avoid draining battery or mobile data, which would generate user complaints).

• Throttles bandwidth consumption to stay below the threshold of user noticeability.

• Handles NAT traversal to make the device reachable for outbound proxy sessions.

The apps embedding these SDKs span a wide range. Just to name a few:

• Free VPN clients.

• File utility apps.

• Weather apps.

• Games.

So, the SDK is not the app’s core function: it’s the monetization layer that the developer has grafted on.

Opt-In Peer Networks

Some providers operate a peer model, where users knowingly and explicitly enroll their devices in exchange for a direct benefit. This typically is free access to a premium VPN or bandwidth credits. The consent flow clearly explains that the user’s connection will be used to route third-party traffic.

This is the most ethically defensible sourcing model, as the value exchange is explicit and the user retains full awareness of what their device is being used for.

Ethical Considerations

The ethical landscape of residential proxy sourcing is genuinely complex, and it deserves some treatment to help you discern the right proxy provider to choose.

The Consent Problem

The foundational ethical issue is informed consent. When a residential device is used as an exit node, its owner’s internet connection carries traffic for a third party. This means that their IP address becomes associated with requests they did not initiate, to targets they may never have interacted with. If a scraping client uses that exit node to send requests that the target considers abusive, the device owner’s IP is the one that appears in server logs. The potential consequences for an unknowing participant range from having their IP blacklisted by services they use personally to, in extreme cases, being flagged by automated fraud or abuse detection systems.

This is why true informed consent in this context is very important. Also, for deep ethical reasons, informed consent would require a clear, plain-language explanation that the user’s internet connection will relay unknown third-party traffic, that their IP will be the visible origin of that traffic, and that this may affect their IP’s reputation.

Bandwidth and Resource Consumption

Enrolling a device in a proxy pool consumes the device owner’s bandwidth. In many markets, this resource is metered and has a real financial cost. In mobile-heavy markets where Wi-Fi and cellular data are tightly managed household resources, this is a non-trivial extraction of value from the device owner. This is another reason why proxy providers offer rewards to IP owners.

The Downstream Use Problem

Even when sourcing is fully and clearly consensual, the ethical chain does not end at the exit node: it extends to what the proxy traffic is used for. This is because a residential proxy provider cannot fully control what its clients scrape. This means that the same infrastructure used by a legitimate price monitoring service can be used for credential stuffing attacks, ad fraud, scalping bots, or scraping protected personal data.

For this reason, responsible providers implement terms of service enforcement, traffic monitoring, and client vetting. As a scraping professional, you are a link in this chain and carry some responsibility for the downstream ethics of the traffic you generate.

Legal Exposure

The legal status of residential proxy usage varies by jurisdiction and use case. In the United States, the Computer Fraud and Abuse Act (CFAA) has been interpreted broadly to potentially cover unauthorized access to computer systems. And whether bypassing a website’s IP-based blocks via residential proxies constitutes “unauthorized access” has been the subject of significant litigation.

In the European Union, the GDPR introduces additional complexity when scraped data includes personal information. So, note that using residential proxies does not insulate you from these legal frameworks. It only changes the network-layer attribution of the requests.

What to Look For in a Residential Proxy Provider

If you’re new to the scraping industry and need to get residential proxies, you may be overwhelmed by the offers that are on the proxy market. There are a lot of providers out there, and choosing the best one for you may be hard.

If you want to evaluate providers not only based on budget, the suggestion for you is to primarily make ethical considerations. If you don’t know where to start, indicators of a responsible residential IP sourcing model include the following:

• Published transparency reports detailing how exit nodes are recruited.

• Clear opt-in documentation and publicly accessible SDK disclosure pages.

• Active terms of service enforcement against prohibited use cases.

• Direct compensation models for exit node participants.

• Compliance certifications. Some of the most ethical providers have pursued SOC 2 or ISO 27001 certifications, which indicate a structured approach to operational governance.

Conclusion

Residential proxies are easy-to-use tools, but may be hard to understand. The gap between “it works” and “I know why it works” is where most engineers live, and it’s fine until it isn’t.

Understanding the full stack changes how you make decisions for your scraping projects. Why? Because it changes how you pick a provider, how you evaluate proxy types for a given target, and how you think about the ethical and legal weight of the infrastructure you’re routing traffic through.

So, now that you know how this infrastructure actually works, let us know: does it change how you’d pick proxies for your next scraping project?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

I’m picking up where I left off in a previous investigation, so I won’t repeat all the details here. Back then, I pulled libshopeeaegis.so (one of Shopee’s native crypto libraries) into Ghidra and rebuilt it in Python, byte for byte. That was a straightforward win: just an integer switch over standard AES, SHA, and HMAC, once you get past the intentionally confusing symbol names. But as I mentioned at the end of that article, Aegis is just a general-purpose toolbox, not the actual library that signs API requests. The real signer is a different beast, and that’s what I’m looking at now.

What actually gates the API

Capture any Shopee API request, strip the anti-fraud headers, replay it, and the backend answers with an HTTP error and code 90309999. That code is the whole story of this article. It is the anti-fraud rejection, and by the end you will have seen it come back from the app, from a plain HTTP client, and from two different real browsers.

Two headers carry the protection. On the Java side, two interceptors in okhttp, the HTTP client library the Android app is built on, attach them: com.shopee.app.network.antifraud.b and .d, which name themselves SecurityNewSapInterceptor and SecurityNewSapPostInterceptor. The first header is af-ac-enc-sz-token, a constant for the session:

af-ac-enc-sz-token: ghHLT8yNv0Xd8t9C2/t6Sw==|uJEC5q...JfDCw4ImZS5+gWyOO5Dr...8bd|rsGw3AWMR7Ryh7Jo|08|1

The right foundation to keep a smooth scraping workflow. anyIP helps teams collect web data more consistently with high-quality residential and mobile IPs worldwide.

Explore anyIP Proxies →

This token is a pipe-delimited string: an encrypted device fingerprint, a 16-byte wrapped key, an 81-byte ciphertext, a 12-byte nonce, and two version markers. It’s computed once per session and then reused. The second header, x-sap-ri, comes with a set of signatures keyed by short hex strings, and these change every request since they’re computed over the request itself. Both headers are generated by libshpssdk.so, a hefty 4.7 MB native library. The tricky part is the per-request signature, since that’s the bit you can’t just precompute and reuse.

Finding the signer, and deciding not to port it

libshpssdk.so exports exactly one symbol, JNI_OnLoad. Everything else is hidden, which tells you the native methods are registered at runtime with RegisterNatives, the Java Native Interface (JNI) call an app uses to bind native functions to Java methods, rather than exported by name. So we catch the registration as it happens. On arm64, RegisterNatives is a slot in the JNIEnv function table at byte offset 0x6b8. We hook that slot spawn-gated, before the app’s own code runs, and read out every method the library registers. The agent is in /shopee_sap_oracle/agent_oracle.js.

We are after one JNI signature. A request signer takes the request and returns headers, which in JNI terms is two byte arrays in and a string out, ([B[B)Ljava/lang/String;. One method matches. It is vuwuuwvw, registered at libshpssdk+0xc7ebc, on the obfuscated class com.shopee.shpssdk.wvvvuwwu, a name we recovered from the jclass argument of the registration call through java.lang.Class.getName().

When I opened that address in Ghidra, I found the function itself barely does anything. It just marshals its arguments and passes them to another function, along with a 440-byte blob and a table of function pointers. That’s the classic shape of a bytecode interpreter: the blob is the program, the pointer table is the opcode handlers, and the real logic lives in the bytecode, not in any native function I can easily read. This is where my earlier approach hits a wall. Last time, I could just read the primitive bodies and match them to AES and SHA. Here, there are no such primitives. To reimplement the signer, I’d have to reverse and interpret the whole virtual machine (VM), a project that would take weeks and is on the same level as breaking DroidGuard, Google’s own anti-abuse VM. So instead of porting it, I decided to just call it directly.

The one thing that works: the oracle

The signer is a Java method on a class we already resolved, so we drive the app’s own code. The agent exposes one RPC (remote procedure call) export that takes a URL and a body, turns them into the two byte arrays the method expects, and returns whatever the signer returns. The driver (/shopee_sap_oracle/frida_oracle.py) spawns the app, waits for the signer to register, and signs requests we choose. Re-running it against the current app, the signer registers in three seconds and answers:

[SIGN] url=https://mall.shopee.ph/api/v4/native/homepage body={"test":1}
   3c7b6059 = LR4te3nzTKb0RcbDqGs1bsXb0pB=
   71356940 = uMaRB6hpxBCq1BwZ2bFqGXjGFow=
   a0e0dcb3 = WFiutLJdM2C0t5Lr4yaoPr9CLn4=
   eaab58ec = 3BJOkkGhS53kj19RA2QqVLfC1m8sZFLSYm8W8S+Kl3yHU+vuzvb+...   (~2 KB)
   x-sap-ri = d9e7576a26efc4c9896e6716017c0e63cf0cae2c4c560ed7dd03

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

That’s the win, and I want to be clear about what it actually means. I can now produce the exact headers the app would attach for any request I choose, without needing to understand how the signer works under the hood. But before I trust these headers, it’s worth looking closely at the output, because it gives you a sense of how tough it’ll be to reuse them. The structural checks I’m about to show run on the captured values using /shopee_sap_oracle/analyze_headers.py. No device needed.

The short values look like base64 of a 20-byte digest, which is the length of HMAC-SHA1, so the easy reading is that they are keyed hashes and that is where a first pass leaves them. The easy reading is wrong. Decode seven of them as standard base64, and five come back with non-zero bits in the final padded position, which a conforming base64 encoder never produces from raw bytes:

  6m6r94xGLKAjTMYNMepdMiLQpSB=  len=20  NON-canonical (low2=1)
  wLJO02yz2I+FAZETVwC0aTCflyB=  len=20  NON-canonical (low2=1)
  1iwMbNiaPma2XLT7ToRs4jgiasE=  len=20  canonical
  47WhCSbrTx1r5FdFFlgtpHPN0OO=  len=20  NON-canonical (low2=2)
  LzCPR2/LYpvqP/z1HBXtuzj+8dQ=  len=20  canonical
  vmPAmagMmOlffOGIWF8MwJEBuNT=  len=20  NON-canonical (low2=3)

Check the TWSC YouTube Channel

So these aren’t just plain base64-encoded digests. Either the alphabet is shuffled or there’s some transformation after encoding, and that’s exactly the kind of detail that trips up anyone trying to reproduce the signature with HMAC-SHA1 and a standard encoder. The keys themselves are interesting too: within a session, no two requests share a key, and those eight-hex identifiers aren’t the actual header names you see on the wire. Plus, endpoints with a body return four entries, while GETs without a body return three. There’s a whole mapping layer from the signer’s per-request keys to the real header names, and I never managed to reverse that part.

Then there is x-sap-ri itself. It is 26 bytes: a leading counter, a three-byte session ID, an eight-byte middle field, a 01 marker at byte twelve, and a thirteen-byte tail. The counter is the interesting part, because it is coarser than it looks. In the fresh capture, both requests came back with counter d9 and session e7576a in front, and only the middle and tail differed:

d9 e7576a 26efc4c9896e6716 01 7c0e63cf0cae2c4c560ed7dd03
d9 e7576a b4c3b40323380413 01 d56d3f5c7eaef45395425b9720

So the leading byte isn’t just a simple counter that increments with each request. What actually separates one request from another is hidden in the middle and tail fields, and the token is tightly bound to the specific request through those parts. That binding is the first reason why a replay fails, and it’s where the real obstacles begin.

App wall one: the replay is out of sequence

The obvious next step was to sign a real product request using the oracle and try replaying it from outside the app. I picked a phone case from Shopee’s public site (shop_id=20396998, item_id=5420033304), had the oracle sign its product detail URL, attached the session’s af-ac-enc-sz-token and a cookie I grabbed with a one-shot header hook, and replayed the request with curl-cffi. The driver for this is /shopee_sap_oracle/replay.py. Here’s what Shopee sent back:

[*] oracle signed 4 headers: ['5cfcca02', 'c382fc01', 'cf1e6149', 'x-sap-ri']
==== REPLAY RESULT ====
HTTP 403
{"5":false,"2":false,"0":2,"3":90309999,"error":90309999, ...}

The status code changes depending on the endpoint, but the error code is always the same. Having a valid signature is necessary, but it’s not enough. This replay was always going to fail for two reasons: first, the x-sap-ri binding ties the token to the device’s own request sequence, so any injected, out-of-sequence request gets rejected, even with a fresh signature. Second, I put the signer’s per-request-keyed values under those same eight-hex keys, but I never actually confirmed that’s how the app names them on the wire. For a clean external replay, you’d need both of those right, and I could only be sure about one.

App wall two: the RASP cancels everything

The way to remove both doubts is to stop replaying and make the request from inside the app, where the sequence, the device state, the TLS, and the real header names are already coherent. The app builds its API calls on okhttp, so we went looking for its OkHttpClient at runtime with Java.choose. The probe is in /shopee_sap_oracle/crash_trace.js. There are more than a dozen live instances, and you spot the API client by its interceptors. The antifraud layer is not among the application interceptors where we first looked. It sits lower, among the network interceptors, which run after okhttp has set up the real connection, so it signs the request that is actually about to go on the wire.

As always, the code that will be used for the python reimplementation we’re showing now can be found in our GitHub repository reserved for paying users, inside the folder 110.SHOPEE-PART2.

Read more

If you’ve run a scraper at any real scale recently, you’ve felt the shift. The block didn’t come from too many requests per minute, and rotating to a fresh IP didn’t fix it. The page just quietly refused to load the data you came for. That’s fingerprinting doing its job, usually before your first byte of useful content arrives.

We keep hitting the same pattern. A pipeline that ran fine for months starts returning empty pages or CAPTCHAs, the proxy pool looks healthy, the rate limits are nowhere near tripped, and nothing in the logs explains it. What changed is not how many requests you send. It is how much the target now knows about the client sending them.

What browser fingerprinting is, and why it matters for scraping

Fingerprinting identifies and tracks a client from the unique combination of characteristics its browser and device expose, with no cookies, no login, and no explicit identifier required. It doesn’t ask “who is this user?” It asks “does this look like the same client as before, and does this client look real?”

The signals are individually boring. Browser version, OS, screen resolution, installed fonts, timezone, locale, how the device renders graphics. None of them identify you alone. Put them together and you get a profile distinctive enough to survive across sessions, IP changes, and incognito mode. For the defense side that’s a bargain, because one collection step feeds fraud detection, bot detection, rate limiting, and access control at once.

From IP-based blocking to identity-based detection

The traditional model was simple. Send requests, rotate IPs, stay under the rate limit. As long as each request looked like it came from a different address, you stayed under the radar.

Fingerprinting breaks that model cleanly. Rotate all the IPs you want, and if your browser fingerprint stays identical across them, you’ve just told the anti-bot that one client is hiding behind a dozen addresses. No real user does that. It is more suspicious than a single sticky IP would have been. The rotation you relied on becomes the tell.

How fingerprinting works under the hood

Modern systems don’t lean on one identifier. They build a layered profile across browser properties, device capabilities, rendering behavior, and network characteristics, then score how coherent the whole thing looks. The more consistent and ordinary the profile, the more trust it earns. That last part is the one most scrapers get backwards, and we’ll come back to it.

Passive signals

These come for free, straight from the HTTP request or a basic JavaScript query, with no heavy lifting required:

• User-Agent string (browser, version, OS)

• Accept headers (language, encoding support)

• Screen resolution and color depth

• Timezone and locale

• Installed fonts and plugins

Active signals

Active fingerprinting runs JavaScript to pull things that are much harder to fake, because they depend on how the hardware actually behaves:

• Canvas fingerprinting renders hidden graphics. Tiny differences in font rasterization, anti-aliasing, and GPU instructions produce a distinctive output.

• WebGL fingerprinting probes the GPU and surfaces details about the graphics hardware and driver stack.

• AudioContext fingerprinting reads the audio processing pipeline, which varies slightly across systems and adds more entropy.

These matter because they measure behavior, not claims. A User-Agent is just a string you can set to anything. A Canvas hash is the device answering for itself. Spoof the User-Agent but leave the Canvas output screaming “headless Chromium on a Linux server,” and the two halves of your story don’t match.

Network-level signals

Detection doesn’t stop at the browser. The network layer sanity-checks the rest of the profile:

• IP address and ASN (who owns the network)

• Geolocation consistency, checking IP against timezone against language

• TLS fingerprinting (JA3, and increasingly JA4), the shape of your TLS handshake, which most HTTP clients get wrong in a recognizable way

• HTTP/2 fingerprinting, the frame and settings ordering that Akamai in particular leans on

• Connection reuse and timing patterns

A browser claiming to be mobile Safari in Spain, connecting from a datacenter ASN with a Python-default TLS handshake, has three independent signals contradicting each other. Any one is forgivable. Together they’re a verdict.

What makes a fingerprint suspicious

Most fingerprints aren’t obviously fake. They’re built from real values: common browsers, standard resolutions, familiar operating systems. What anti-bot systems hunt for is how those real values fit together.

The first giveaway is a mismatch between attributes. Real devices follow predictable patterns. Certain browsers cluster on certain operating systems, resolutions cluster around popular hardware, fonts and rendering align with the platform. A macOS Safari profile serving Windows fonts, or a mobile User-Agent reporting a 2560-wide desktop viewport, breaks that pattern. Each discrepancy is explainable on its own. Stacked together, they aren’t.

Then there’s rarity. A perfectly unique fingerprint feels like good privacy and is terrible for blending in. Real users fall into large, overlapping groups of near-identical configurations. A profile built from attributes that almost never co-occur is both easier to track and quicker to flag. This is the over-randomization trap, and it’s the most common self-inflicted wound we see. People randomize every parameter and end up with a device that has never existed on Earth.

Consistency over time counts just as much. Real users are stable, same device, same browser, broadly the same setup from one request to the next. When one fingerprint shows up on an IP in one region and then again on an IP in another, or when key attributes drift between sessions, it reads as synthetic. Several IPs sharing a single identity is a pattern normal traffic just doesn’t produce.

Automation leaves its own tells. Even a carefully configured environment leaks something: a missing or stubbed API, a navigator.webdriver flag left on, slightly different JS execution paths, rendering quirks that don’t match a real build. No single tell is fatal. Detectors collect the small ones until the picture tips toward automation.

Last comes behavior. A flawless fingerprint still falls apart under robotic behavior. Pages loaded instantly with no intermediate navigation, clicks landing with pixel precision, sessions that fire one request and vanish, all of it undercuts the identity. Online, your identity is as much how you move as how you look.

Why fingerprinting breaks the old playbook

For years the routine was simple. Rotate IPs, randomize headers, respect rate limits. It worked because defenses were built around request-level signals, mostly where traffic came from and how fast it arrived. Fingerprinting moves the decision from the isolated request to a persistent identity, and most of the old tricks stop paying off.

Stateless requests became stateful identities. The old model judged each request on its own. Block one IP, grab another, keep going. Fingerprinting adds continuity. Change the IP and the browser environment can still look identical, or inconsistent in ways that link the sessions together. Rotating IPs no longer resets who you are.

Header randomization stopped working too. Rotating User-Agents and tweaking Accept-Language used to add diversity. Fingerprinting doesn’t read those values in isolation. It checks whether they agree with each other. Spin up a new User-Agent on every request while your rendering and JS execution stay constant, and you don’t look like many users. You look like one unstable client lying about itself. Tools like Apify’s fingerprint-suite and BrowserForge exist because building a coherent set of headers and fingerprint values is harder than randomizing them.

Datacenter-only setups draw scrutiny. Datacenter proxies are fast, cheap, and easy to rotate, but once the network is part of the identity that stops being free. A datacenter ASN has a different shape than residential traffic, and pairing one with a fingerprint that claims to be a consumer laptop is a contradiction. It won’t always block you, but it raises the baseline scrutiny, and stacked with any other inconsistency it pushes you over the line. This is why residential and mobile proxies have become table stakes on harder targets, even though they only fix the network half of the problem.

Detection also moved earlier. Older systems often gave you partial access before a rate limit kicked in. Now the fingerprinting script usually runs the moment the page loads, sometimes before any real content is served. Fail the check and you never reach the data. You get a CAPTCHA, a soft block, or a clean-looking empty response.

How to build a setup that survives fingerprinting

The goal isn’t to be invisible. It’s to be unremarkable, an identity where every layer tells the same ordinary story.

Run a real browser engine

Lightweight HTTP clients can’t fake Canvas, WebGL, or AudioContext, because those depend on a real rendering pipeline. So most serious setups drive an actual browser with Playwright, Puppeteer, or Selenium.

Vanilla automation still leaks, which is where the patched ecosystem comes in. playwright-stealth and puppeteer-extra-plugin-stealth paper over the best-known tells. undetected-chromedriver and its async successor nodriver harden Selenium-style automation, and patchright is a drop-in patched Playwright aimed at the same goal. When you need full managed profiles, anti-detect browsers like Camoufox (open-source, Firefox-based, with fingerprint injection built in), Kameleo, GoLogin, Multilogin, AdsPower, and Dolphin Anty handle the consistency for you. None of these is a magic bullet, since anti-bots study them too, but they clear the low-level signals that plain Playwright fails on day one.

Keep each fingerprint internally consistent

This is where over-randomization kills more scrapers than under-randomization ever has. Don’t change every parameter on every request. Give each session one stable, coherent identity, where the User-Agent matches the OS, the resolution matches the device class, and fonts, language, and timezone all tell the same story. If you generate fingerprints programmatically, lean on something built for coherence, like BrowserForge or fingerprint-suite, rather than rolling your own random values.

Align the fingerprint with the network

A fingerprint is never judged alone. Claim to be a mobile user in Spain while connecting from a datacenter in another region, and the whole session weakens. Treat proxies as part of the identity layer, not just a source of IPs:

• Residential or mobile proxies to match real-user network conditions

• Geo-targeted IPs so the IP, timezone, and language reinforce each other

• Session-based (sticky) routing for continuity across a session

And don’t forget the handshake. If your browser fingerprint is flawless but your TLS fingerprint is a Python default, JA3/JA4 will give you away. curl_cffi, curl-impersonate, and tls-client exist to make that layer match the browser you’re pretending to be.

Treat sessions as isolated identities

Each session should behave like its own user, with its own fingerprint, its own IP, and its own cookies and local storage. Don’t reuse one fingerprint across several IPs at the same time, and don’t let session state bleed between identities. In practice that means pairing one fingerprint to one IP for the life of a session, then rotating both together, the way a real device and its connection move as a unit.

Make the behavior match the identity

A perfect fingerprint can’t save robotic behavior. Real users don’t teleport onto deep pages, strip the data, and disappear. They land somewhere shallow, navigate, pause, scroll. Adding delays that reflect reading time, and routing through intermediate pages instead of hammering endpoints directly, does more for credibility than another round of fingerprint tuning.

Know when to stop building it yourself

Maintaining all of this is real, ongoing work, and the targets keep moving. Browser hardening, fingerprint coherence, proxy alignment, session isolation, behavioral pacing, every layer needs upkeep. At some point the maintenance cost outweighs the control you get from it, and pushing these layers down into infrastructure starts to make sense.

That’s the niche a managed scraper API fills. Instead of hand-coordinating proxies, browser environments, and anti-detection logic, you hand off the identity layer and keep your code about the data. Infatica’s Scraper API is built for that hand-off, with fingerprinting, session management, and request routing handled underneath, so what you maintain is extraction logic rather than evasion logic. It isn’t the right call for every project, but when anti-detection upkeep starts eating your roadmap, it’s a trade worth weighing.

Where this actually bites

Fingerprinting decides what data you can reach, how trustworthy it is, and whether your pipeline survives at scale. Getting past IP filtering is step one. Holding a believable identity for the whole session is the real game, and it’s where most of the projects we see actually break.

Price intelligence and travel feel it first. Travel platforms are among the most aggressive adopters. Prices move by location, device, and browsing pattern, so a setup that trips the defenses sees prices real users never see, or different ones every session. A hotel-pricing comparison is only as reliable as the identities behind the requests.

Search results are another. Search engines read the context of a query, not only the query itself. Weak fingerprinting earns an instant CAPTCHA. To capture what real users in a given region actually see, the requests have to look like they came from real users in that region, on consistent devices.

E-commerce monitoring runs into the same wall, and we’ve watched it happen most often on the luxury retailers sitting behind Akamai. Retailers tune availability, pricing, and promotions by user profile, and they use fingerprinting to tell shoppers from bots. Get flagged and you see hidden or reordered listings, throttling, or temporary bans, small distortions that quietly poison competitor-pricing analysis over time.

Ad verification and market research depend on seeing exactly what a real user sees: ad placement, localized copy, device-specific variants. Fingerprinting interferes by serving fallback content, blocking certain placements, or masking geo-targeted campaigns. That’s fatal when you’re validating campaigns across regions.

Account-based platforms lean hardest on fingerprinting to spot unusual access. These setups tend to fail not on a single request but because the session itself stops looking trustworthy. A sudden device-profile shift, an IP-location jump, or off behavior can each trigger a security check or kill the session.

Conclusion

Fingerprinting changed the question. It used to be “where is this request coming from?” Now it’s “does this client hold together as a believable identity, request after request?” Browser attributes, network signals, and behavior all feed that judgment, and the old reflex of rotating IPs and randomizing headers actively works against you when the layers don’t agree.

The setups that hold up aren’t the ones that randomize the hardest. They’re the ones that look the most ordinary: one coherent fingerprint, matched to one plausible network, behaving like a real person, held steady for the length of a session. The closer your setup sits to how real users actually browse, the more reliable your data collection becomes, and the more boring, in the best way.

Fortunately, not all providers have gone in that direction (think of the simple one-click Cloudflare Turnstile). However, many others are designing more intricate mechanisms to counter AI-based automation.

Is this the end of CAPTCHA solvers? And does this impact your scraping script? And how should we interpret these changes overall? Let me introduce you to the new generation of CAPTCHA systems to address all of these questions!

Give your AI a web data layer – Decodo's Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

Why CAPTCHAs Have Become More Sophisticated

Before the rise of modern AI, CAPTCHAs could already be solved automatically using traditional machine learning techniques and browser automation. The approach was based on three steps:

1. Detect the challenge.

2. Recognize its content.

3. Simulate the required user interaction.

For example, image classification models can identify the correct images in reCAPTCHA v2 (”Select all images containing hydrants”):

(For more details on reCAPTCHA solving, see our previous articles: Part I and Part II.)

Similarly, OCR models could extract text from distorted image-based CAPTCHAs.

Combined with browser automation, these models were often accurate enough to solve many CAPTCHA challenges without human intervention.

A good example was Botright, an open-source browser automation library that integrated machine learning models to solve CAPTCHAs. The project has since been abandoned (its last commit was over two years ago), but it demonstrated how effective this approach could be.

I’ve personally tested similar techniques. While they were far from 100% reliable, they were still capable of passing many CAPTCHA challenges.

CAPTCHA providers have long been aware of these techniques. As automated solvers became more capable, they responded by making CAPTCHAs steadily more intricate. This led to the introduction of puzzle-based challenges, such as those used by AWS WAF CAPTCHA, hCaptcha, and other providers.

Yet, the rapid advancement of multimodal AI models has raised the bar even further…

Instead of training specialized ML models for each CAPTCHA type, LLMs can understand images, reason about what they see, and let the controlled browser interact with the page much like a human would. That has forced CAPTCHA providers to develop more sophisticated techniques to distinguish humans from bots.

This is where we stand today!

Are CAPTCHAs Dead Because of AI?

The short answer is: No!

While multimodal AI models have fundamentally changed how CAPTCHA providers think about distinguishing human users from automated bots, traditional CAPTCHAs are far from obsolete. There are at least three reasons why:

1. Not every bot uses AI: Many scraping bots still rely on traditional browser automation or simple scripts. They don’t have access to AI tools capable of understanding and solving CAPTCHA challenges.

2. LLMs are not deterministic: AI models can solve the same CAPTCHA challenge most of the time, but fail spectacularly on others. No wonder you can find examples on Reddit and X of AI-powered browsers repeatedly failing at CAPTCHA challenges.

3. Solving the challenge is only part of the verification process: CAPTCHA systems usually evaluate much more than just the answer to the test. They also analyze behavioral signals such as mouse movements, cursor acceleration, click timing, scrolling behavior, touch interactions, and many other indicators. That’s why one-click, press-and-hold, and slider CAPTCHAs exist. An AI model may correctly identify the solution, yet still exhibit interaction patterns that look “robotic” rather than “human,” causing the verification to fail.

That said, AI is undeniably making life harder for CAPTCHA providers. Numerous research papers have shown that, given sufficient training data and realistic browser automation techniques (such as human-like mouse movements, scrolling, and timing), AI systems can successfully solve many traditional CAPTCHA schemes.

The real question is no longer whether AI can solve older CAPTCHAs, as it clearly can. The more interesting question is: Can it reliably solve the new generation of AI-resistant CAPTCHAs?

New Approaches to AI-Resistant CAPTCHAs

More and more CAPTCHA providers are experimenting with advanced challenges specifically engineered to resist AI-powered solvers.

In this section, I’ll focus on three approaches to newer CAPTCHAs that I’ve personally seen challenge even some of the most popular CAPTCHA-solving solutions.

hCaptcha’s New Puzzle-Based Challenges

hCaptcha has always been one of the more difficult CAPTCHA providers to bypass, thanks to its creative human verification tests.

As AI capabilities have advanced, hCaptcha has responded by introducing elaborate puzzles purpose-built to thwart automated solving:

Notice how many of these challenges require much more than simply clicking on the correct images. They can involve drag-and-drop interactions, spatial reasoning, and specific knowledge.

Instead of fighting every CAPTCHA, focus on avoiding them. anyIP gives you access to high-quality residential and mobile proxies that help your traffic blend in more naturally.

Start with AnyIP

Microsoft’s Multi-Option CAPTCHAs

Explore the Microsoft forums and Reddit, and you’ll find plenty of users complaining about Microsoft’s newer human verification challenges (example 1, example 2, example 3). Here are a few examples:

These challenges can be somewhat ambiguous. They tend to present multiple plausible answers, require you to search through numerous options, and may involve several consecutive steps before the verification is complete. As a result, they take considerably longer to solve and can be frustrating for legitimate users (especially if a mistake forces them to restart the entire challenge).

Temu’s Proprietary CAPTCHAs

Some platforms are moving away from standard CAPTCHA providers and building their own verification systems in-house. A notable example is Temu, which has recently introduced a restrictive, custom CAPTCHA mechanism:

These challenges go beyond simple image selection and require very specific actions. For instance, they can involve multi-step interactions such as drag-and-drop tasks or slider-based tests that must be completed in a precise way.

What they generally have in common is the heavy use of AI-generated images. This creates an interesting dynamic: AI is being used to produce the very content to stop AI-powered bots.

The Future of CAPTCHA: What Comes Next

While these new CAPTCHA challenges are definitely more difficult for AI to solve, it’s still unclear whether AI will evolve quickly enough to keep up in the ongoing arms race between scraping developers and CAPTCHA-protected websites.

What’s clear is that providers will continue experimenting…

Hand-Gesture CAPTCHAs

Hand-gesture CAPTCHAs represent a shift from traditional image-based verification toward real-world interaction signals.

Google has recently begun testing a new reCAPTCHA system that asks users to prove they’re human by performing simple hand gestures in front of their device camera.

When enabled, the browser requests camera access and records a short video of the user’s hand movement. The system then analyzes the footage by extracting around 21 hand-landmark points (measurements of finger and joint positions) to identify specific gestures.

This is an attempt to improve “liveness detection,” making it harder for automated systems to pass as real users. Still, it’s not impossible that AI-driven tools could eventually simulate these gestures using AI-generated videos via camera injection.

Check the TWSC YouTube Channel

Stronger CAPTCHAs, Worse User Experience?

Newer CAPTCHAs are significantly harder for AI to pass, but they also introduce a new layer of friction for us, the everyday users. That’s the main trade-off in the current wave of anti-AI verification systems.

As models become more capable, CAPTCHAs are forced to evolve in parallel, often becoming more demanding not only for bots but also for legitimate users.

As a result, many users have been complaining on Reddit and X about frustration, increasingly time-consuming challenges, and privacy concerns (such as having to use a camera just to complete a CAPTCHA).

This has led to growing criticism that could eventually push providers to relax some of these verification mechanisms, either by making them easier or less frequent (which would be a win-win scenario for web scraping).

It’s quite clear that the real objective is finding a balance between effectively blocking bots and avoiding unnecessary friction for legitimate users.

Are These New CAPTCHAs Effective Against AI?

These newer CAPTCHAs aren’t becoming more elaborate simply for the sake of increasing difficulty. Providers are well aware that the probability of AI reliably passing these checks at scale remains low, though not zero.

The idea is to design challenges that are intricate enough to deter automation, while still incorporating sufficient interaction signals and behavioral data to assess whether the user is actually human.

In essence, beneath the surface, the core principle hasn’t fundamentally changed. What we are seeing is an evolution in execution, rather than a reinvention of the underlying idea.

Need help with your scraping project?

Avoiding CAPTCHAs Remains More Effective Than Solving Them

The final question is: “Should these new CAPTCHAs scare us, web scraping experts/enthusiasts?” In a sense, they might, but ultimately this is not really a revolution…

The truth is that in web scraping, the most effective strategy is rarely about solving CAPTCHAs. Instead, it’s about avoiding them altogether.

CAPTCHA systems are just one layer of a broader anti-bot ecosystem, and triggering them is a signal that the traffic is already considered low-trust or detectable.

For this reason, the focus should be on prevention rather than resolution. Best practices involve:

1. Routing traffic through high-quality residential proxy networks with strong reputation IPs.

2. Using stealth browser environments such as Kameleo Docker, Camoufox, or Rayobrowse.

3. Managing and rotating browser sessions in automated scraping workflows.

The end goal is to consistently emulate real, human-like browsing behavior to minimize the chances of detection. If human verification tests still appear, CAPTCHA solvers will remain the go-to option, as they must evolve to keep up with new challenges in order to stay in business.

The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Conclusion

In this post, you’ve understood why CAPTCHAs have become more sophisticated and the role AI has played in this evolution. You’ve also seen what providers like hCaptcha, platforms like Microsoft, and sites like Temu are doing to defend against AI-powered automated scrapers and bots, as well as what the future of CAPTCHAs might look like.

I hope you now have a clearer understanding of the current and future CAPTCHA landscape. If you have any questions, feel free to leave them in the comments below. Until next time!

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

On July 2nd, the FBI, together with several industry partners, seized hundreds of domains tied to NetNut. For those who don’t know, NetNut is a large residential proxy service run by Alarum Technologies, a company that’s even listed on NASDAQ. The seized domains included netnut.com, proxyjet.io, and divinetworks.com.

This didn’t come out of nowhere. Let me walk you through how we got here, because the timeline is important to understand what happened.

How the takedown happened

It all started on June 19th, when three security firms published reports showing that NetNut’s residential proxy network was actually powering a botnet called Popa. The software was running on everyday devices like smart TVs and streaming boxes. Qurium, one of the security firms, stumbled onto this while looking into some denial-of-service attacks and traced the control servers back to pirated streaming apps. Then Brian Krebs made the story public, and about two weeks later, the authorities stepped in.

The takedown was a team effort: Google, the FBI, Lumen Technologies, The Shadowserver Foundation, and others all played a part. This is where things get interesting from a technical perspective, because taking down a proxy network is nothing like just unplugging a server.

Residential proxy networks are built to be decentralized. The exit nodes are actually millions of consumer devices scattered all over the world. Google estimated that NetNut had at least 2 million devices in its network. Obviously, you can’t just go and seize someone’s smart TV. What you can do is target the things those devices rely on to function as part of a commercial proxy service.

First, there’s the command and control layer. Google shut down the accounts and services that NetNut was using to control the malware. What I found interesting is that this C2 infrastructure was actually running on Google’s own services. This technique, called ‘living off trusted services,’ lets malicious traffic blend in with normal cloud traffic. Without command and control, there’s no way to route customer traffic through those devices.

Next is the enrollment pipeline. Google Play Protect began warning users and disabling apps that included NetNut SDKs, and it will keep blocking any new installs. If a proxy network can’t add new devices, it starts to shrink over time, since devices get reset, replaced, or just go offline.

Then there’s the commercial front door. By seizing the main domains, the FBI basically shut down NetNut’s storefront and made it impossible to bring in new customers at scale.

All of this together, according to Google, cut down the number of available devices by millions and seriously hurt NetNut’s business.

What makes this case different

I’ve seen proxy networks get taken down before, but NetNut is different. This wasn’t some anonymous group hiding on the dark web. NetNut was run by a company listed on NASDAQ, with investor relations, quarterly reports, and even a legal team that put out a statement saying they were aware of the seizure and cooperating with investigators.

It must be said that Alarum has rejected the accusations, disputing their accuracy. On the commercial and official side, NetNut has always presented itself exclusively as a legitimate proxy provider for enterprise data collection, without ever declaring any kinship with the clandestine Popa botnet. In its communications following the seizure, the company stated that neither Alarum nor NetNut had been formally contacted by the FBI or any other governmental authority in connection with these matters, while temporarily suspending data traffic through the affected network services in order to investigate the incident and assess the infrastructure involved. Markets, however, delivered their own verdict quickly: Alarum’s stock crashed by more than 70% at the end of last week, bringing its market cap down roughly 96% from its 2024 peak, and the company itself warned that prolonged disruptions are expected to have a material adverse effect on its operations and financial results.

It’s important to be clear about which part of NetNut’s network is involved here, because NetNut wasn’t just selling one type of proxy. They offered rotating residential proxies, static ISP proxies, mobile proxies, and regular datacenter proxies. Datacenter IPs are the least controversial, since they come from cloud and hosting providers with clear ownership. The takedown focused on the residential side, where NetNut actually used two very different sourcing models.

On one side there are the ISP proxies, sourced through DiviNetworks, the company NetNut originally grew out of. This is an above-board arrangement: as Spur documented in its research on how proxy providers co-opt entire networks, DiviNetworks recruits ISP partners with a straightforward revenue-stream pitch, terminates a GRE tunnel on the partner’s edge router, installs software on a VM inside the partner network, and uses policy-based routing to steer a slice of traffic through that VM and back out through NetNut’s infrastructure. From the partner’s perspective, a small amount of traffic is diverted; from NetNut’s perspective, selected egress from the partner’s public IP space becomes a commercial proxy product. The ISPs sign an agreement and get paid. You can argue about whether their subscribers know, but at least the contractual chain is in daylight.

The other side is the residential proxy pool, and this is where the Popa botnet comes into play. Researchers found that NetNut built this network by distributing SDKs for devices like smart TVs and streaming boxes. Synthient looked at over 20 real Popa publishers and didn’t see any of them asking users for consent. This is the part of the network that led to the takedown. When you have two million devices enrolled without real consent, that’s not a legitimate proxy network; it’s a botnet with a billing system, at least from a law enforcement point of view.

But the fact that divinetworks.com was seized along with the other domains tells me that investigators aren’t making a clear distinction between the different parts of the business. Even providers who think their sourcing is clean should take note.

This isn’t just a one-off event; it’s part of a bigger pattern

If this all sounds familiar, it’s because we’ve seen it before, and not long ago. The NetNut takedown follows right after Google’s disruption of the IPIDEA proxy network back in January 2026. At the time, IPIDEA was one of the biggest residential proxy operations out there, and NetNut’s main competitor.

Google’s post-mortem on IPIDEA included what I think is the key point here: when their own botnet starts to fall apart, proxy operators just buy capacity from competitors and become resellers. The proxy industry is tightly connected, and Google is pretty confident that a lot of popular residential proxy brands were actually just reselling NetNut’s botnet.

If you buy residential proxies from a brand you haven’t really checked out, pay attention. The IP pool behind your provider’s logo might be NetNut’s, or IPIDEA’s, or whoever is next in line.

The timing here is no accident. Bot traffic is exploding, especially with AI agents and LLM-based crawlers on the rise. In all my years doing this work, I’ve never seen web scraping infrastructure get this much attention from regulators and platforms. We’re in a real bind: we need more scraping and automation than ever, which means more proxies, but at the same time, there’s a real push for transparency about how these networks are built and used, so we can tell the good actors from the bad.

The lesson from NetNut is that just being transparent about where your IPs come from isn’t enough. You also need to know who is actually using your network. Alarum said they did KYC and due diligence on NetNut’s customers, but Spur found that NetNut didn’t really require corporate verification or real know-your-customer checks before selling proxy access. Down the line, resellers and white labelers usually didn’t do any KYC at all. Anyone who knew where to look could buy access with just a burner email and a few dollars in crypto. The result? In just one week in June 2026, Google saw 316 different threat groups using NetNut exit nodes, including cybercriminals and espionage actors.

What this means if you were a customer

Let’s be honest: losing a proxy provider isn’t really a technical problem. If you’re running a serious scraping setup, switching providers just means changing your endpoint. And if you’ve set up any kind of automatic fallback system, which you really should have, your pipelines probably won’t even notice the downtime.

The real headache is legal and compliance risk, especially if you’re working in a highly regulated industry.

Take financial markets, for example: I’ve worked in this space myself. Web scraping is a key part of what’s called alternative data: things like scraped prices, app usage stats, satellite images, and transaction data. Investment funds use these to get an edge on listed companies, alongside the usual financial statements and market data.

Anyone who has tried to sell web-scraped data to an investment manager knows what comes next: the due diligence questionnaire. I’ve been through this process, and it’s not a formality. The standard DDQ used in the industry, published by FISD’s Alternative Data Council, dedicates an entire appendix to web scraping and explicitly asks vendors whether they use proxies, whether those proxies are anonymizing, whether the proxy providers include information sufficient for scraped websites to contact the company, whether data is collected from pages behind logins or CAPTCHAs, and whether the vendor or its service providers have received cease and desist orders, subpoenas, or been subject to investigations in the past five years.

Now imagine you’re a data vendor facing that last question, and your proxy provider’s homepage is showing an FBI seizure banner. Telling a hedge fund that your infrastructure supplier is under federal investigation is not going to help you close any deals. In this business, your compliance is only as strong as the weakest link in your supply chain. Proxies are definitely part of that chain, whether your legal team knows it or not.

How I Think About Choosing Proxy Providers After NetNut

The reality is that residential and mobile proxies have always been the most controversial part of the proxy world. Unlike datacenter IPs or ISP deals, residential proxies need real people to agree to run a proxy on their devices in exchange for something, maybe money if they’re in a bandwidth-sharing program, or a free app, since developers get paid by proxy providers for every install.

Most people don’t realize how widespread this is. Spur scanned over 6,000 LG and Samsung smart TV apps and found that more than 2,000 of them had residential proxy SDKs inside. That means about a third of smart TV apps are quietly acting as proxy infrastructure. The real difference between legitimate and abusive setups comes down to consent and disclosure. Some providers, like Bright Data, Massive, and Oxylabs, at least spell out the arrangement in their terms of service, as Spur pointed out. For example, you can check Bright Data's peer program directly on bright-sdk.com, where you can find the certifications and compliance that they obtained.

Some providers have shown they’re willing to take a hit to keep their networks clean. For example, Bright Data has publicly shared that they dropped SDK partners who wouldn’t digitally sign their apps or who served ads leading to shady websites. These partners made up over 10% of their residential peer network, so it wasn’t a small decision. On the You might be skeptical about companies promoting their own good behavior, but I think this is the right direction: sourcing based on real standards, not just growth at any cost. They’re also founding members of the Alliance for Responsible Data Collection, together with other players like Sequentum and Common Crawl.

So here’s what I take away from all this. After what happened with IPIDEA in January and NetNut in July, picking a proxy provider based only on price isn’t just about quality anymore; it’s about risk. You need to ask the same questions the security researchers asked NetNut: How do you source your IPs, and can you prove consent? Who can actually buy access to your network, and what does your KYC process look like, even for your resellers? And what happens, both in contracts and technically, when abuse is found?

If your provider can’t answer these questions, someone else will, maybe with a seizure banner.

You buy proxies from a provider with the biggest IP pool you can find. You rotate IPs for every request. You randomize User-Agent strings. You connect to proxy servers from various regions. You add delays between requests. You expect to smoothly scrape the necessary data in no time, as you have made everything so your scraper does not leave any traces behind and is impossible to get caught. The next thing you see usually happens around request 150-300 - this is blocks. CAPTCHAs, error 403, redirects to honeypot pages serving you wrong data. The whole pipeline crumples down.

If this situation is painfully familiar, you have probably let session stickiness out of sight while building a proxy rotation layer - a common mistake. This article explains in detail how to rotate proxies to mimic real users so your scraper remains a reliable data source even at scale.

Why careless IP rotation may speed up bot detection

When you realize a website has blocked you out, the first instinct is to add more IPs - maybe buy proxies that perform better and are more adapted for dealing with detection systems. You try to add more geo-diversity. In other words, you use the same approach as before, just scale it. Sometimes it helps a bit; however, sometimes, blocks are back even faster.

The mistake here is that you are looking for a problem in the wrong place. It is not the lack of IP diversity, but the lack of session consistency.

Modern anti-bot systems like Cloudflare, Akamai Bot Manager, DataDome, or PerimeterX do not rely on the IP address as a primary method of bot detection. Instead, they consider a lot of signals and model session behavior. In such circumstances, your rotating efforts, if not strategic, can only make things worse and produce behavioral signals that are bot-like, not user-like. More IPs are not a fix for that problem. Modeling behavioral signals of genuine users is.

The anatomy of bot detection

In order to build a truly stealthy scraper with a rotation layer that gets the job done, it is important to understand first how detection happens - what exactly can give you away.

So, there are four layers of different signals that anti-bot systems analyze simultaneously and make a decision whether to let you be or deny access.

• Layer 1 - Network

IP reputation, ASN classification, type of IP (residential, mobile, or datacenter), and provider fingerprints belong to that layer. Also, if a proxy or VPN IP is known, it would also be shown here.

This layer is the one developers usually optimize the most for. Ironically, this is the least important layer for security systems. That is why you fail when following scraping tutorials that offer to simply set as big an IP pool as possible and rotate addresses for every request. It is not because you have done something wrong or your proxies are bad, but because this approach does not work for scraping at scale.

• Layer 2 - TLS/HTTP fingerprints

HTTP clients’ fingerprints are nearly as unique as an IP address. TLS handshake leaves quite a trace with its cipher suite order, list of supported extensions, and elliptic curves. HTTP/2 creates another signal with header prioritization, window size, and frame ordering. The combination of TLS and HTTP/2 fingerprints is often enough for security systems to figure out whether it’s human-generated or bot-generated traffic, even without considering signals from other layers.

Widely used scraping libraries, for example, curl, requests, httpx, aiohttp produce distinct, detectable fingerprints. Python is a popular choice for building scrapers now; still, if you use its requests library, a calibrated WAF would spot it from TLS handshake alone, User-Agent headers be forgotten, whatever they say.

• Layer 3 - Behavioral

Requests cadence, referrer chain integrity, cookie handling, navigation patterns, and timing distributions are evaluated here. Even the delays between requests, if they are too precise and equal, can fuel suspicions, as humans do not make requests with a machine’s precision. Real users’ referrers have logic to them. Human users’ cookies are consistent and show that they came from a search engine page. They do not disappear mid-session or mid-navigation.

• Layer 4 - Session Graph

This layer evaluates whether the sequence of cookies, tokens, page flow, and state transitions within the session makes sense for a human user. That is a layer most scrapes fail to pass, as not so many developers pay attention to coherence. Cause even if stand-alone signals look normal, it is important that they make sense when evaluated together. For example, a user visits an e-commerce website. Their session token would show that. Their viewport size would stay the same throughout the visit. They will go between pages with different time intervals, reflecting reading and decision time.

Coherence is the key that opens or closes a door for your scraper. And one proxy rotation at the wrong time can reveal your scraper’s true identity.

Session stickiness: a true solution

In the context of a proxy architecture, session stickiness is a condition when all requests that belong to the same logical session are routed via the same IP address. At the very least, it must be the same subnet, geo location, and ASN cluster.

Now, let’s define what a session may mean, cause in terms of scraping, there are several options possible.

• Authentication session

The most obvious case. You log in, receive a session token, and proceed to the data. If your IP changes somewhere between the login request and the first data request after authentication, to the target server, it looks like a user logged in from Paris and then suddenly that same user is asking for data from Seoul. Session invalidation or re-verification is guaranteed.

• Stateful pagination

This type is subtler and catches even sophisticated scrapers off guard. The thing is that many modern applications track your pagination state on a server’s side - in other words, a server “remembers” a page you are currently at and ties it to your IP. If the IP address changes between pages 3 and 4, the server loses this track. It may either block you or return incorrect data.

• Multi-step forms

Cases when you need to simulate checkouts, survey scrapers, or lead generation cases are the most session-sensitive. The state that was established at the beginning is carried forward across the rest of the steps. A mid-process IP change is a red flag in such a scenario; detection systems will react immediately.

To sum up, it is not random IPs you need, but sessions and a rotation strategy that considers session coherence.

Building the rotation layer

Time to get to work. Remember: rotating session objects, not IP addresses.

The session object model

A session object, in simple words, is a set of signals that make a set of requests look like they originated from a real human user.

@dataclass
class ProxySession:
    session_id: str
    proxy_endpoint: str          # host:port or provider session token
    cookies: dict = field(default_factory=dict)
    headers_profile: dict = field(default_factory=dict)
    geo: str = ""                # e.g. "pl,us,en"
    asn: str = ""                # e.g. "12345"
    created_at: float = field(default_factory=time.time)
    last_used: float = field(default_factory=time.time)
    request_count: int = 0
    flagged: bool = False

    def is_expired(self, ttl_seconds: int = 600, max_requests: int = 200) -> bool:
        age = time.time() - self.created_at
        return self.flagged or age > ttl_seconds or self.request_count >= max_requests

The headers_profile is not just a User-Agent string. It is a set of HTTP headers that are used for each and every request of one session: User-Agent, Accept-Language, Accept-Encoding, Accept, sec-ch-ua, sec-ch-ua-platform, sec-ch-ua-mobile. They must stay the same for every request. If one request is coming as if from Chrome browser version 108, and another one from version 124, detection systems will notice a mismatch. Even without knowing exactly that you are a scraper, such a mismatch is enough to block you.

Sticky session assignment

Any scraper must use the same session object within one session - this is a rule. So the session assignment must not be random, but based on that rule to prevent any mix-ups.

import hashlib
import threading
from typing import Dict

class SessionPool:
    def __init__(self, proxy_provider):
        self.provider = proxy_provider
        self._sessions: Dict[str, ProxySession] = {}
        self._lock = threading.Lock()

    def _make_key(self, job_id: str, target_domain: str) -> str:
        raw = f"{job_id}:{target_domain}"
        return hashlib.md5(raw.encode()).hexdigest()[:12]

    def get_or_create(self, job_id: str, target_domain: str) -> ProxySession:
        key = self._make_key(job_id, target_domain)
        with self._lock:
            session = self._sessions.get(key)
            if session is None or session.is_expired():
                session = self.provider.new_session(geo=self._infer_geo(target_domain))
                self._sessions[key] = session
            session.last_used = time.time()
            session.request_count += 1
            return session

    def _infer_geo(self, domain: str) -> str:
        # Simplified — in production, map TLDs/domains to geo requirements
        tld_map = {".pl": "PL", ".de": "DE", ".co.uk": "GB", ".fr": "FR"}
        for tld, geo in tld_map.items():
            if domain.endswith(tld):
                return geo
        return "US"  # Default

Notice that the session key is determined by job_id + target_domain, so the scraper sticks to the same session until it expires. The session expiration is also based on 2 rules - TTL (sessions older than 10 minutes automatically get refreshed) and request count (after 200 requests, the session gets refreshed to prevent statistical fingerprinting from long-lived sessions).

Geo-tied rotation

Another life hack is to rotate IPs within the same geo or ASN cluster even when a session has expired, and you are spinning to the new one. Even between sessions, sudden changes (like jumping from Tokyo residential IP to Toronto residential IP) are noticeable and suspicious. When that happens, it looks more like a VPN switch, not a user returning.

Result-oriented proxy providers offer sticky sessions. In such a case, a session token has a location backed into it, and the provider guarantees that the traffic will originate as if from that location during a session’s lifetime.

Header profile coherence

Not only must IP remain the same throughout the session. If you rely on the same address, but the header profile changes from request to request, it is like a “Hello, I’m a scraper” sign for detection systems.

HEADER_PROFILES = [
    {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, br",
        "sec-ch-ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": '"Windows"',
    },
    # Add more profiles reflecting real browser market share distributions
]

One session - one profile. No changes mid-session.

Retry logic

Retry logic is another thing that can blow your cover. When a request fails due to some reason and the target server instantly receives another one from the same origin, it most certainly means bots are in the game, as humans do not act with such light speed. An even less obvious trap is retrying with a different session object when your current session simply needs a brief backoff.

The better approach is to retry while relying on the same session object and a slight delay. If you receive a clear block signal, like 407, 429 errors with specific headers, or CAPTCHA, switch to the new session.

def fetch_with_retry(
    url: str,
    pool: "SessionPool",
    job_id: str,
    target_domain: str,
    max_retries: int = 3,
) -> requests.Response:
    failures: list[str] = []

    for attempt in range(max_retries):
        session = pool.get_or_create(job_id, target_domain)
        try:
            response = make_request(url, session)
            if response.status_code in (407, 429) or is_captcha(response):
                session.flagged = True
                failures.append(
                    f"attempt {attempt}: session {session.session_id} "
                    f"blocked (status={response.status_code})"
                )
                if attempt < max_retries - 1:
                    base = min(2 ** attempt * 2, 30)
                    time.sleep(base * (0.8 + 0.4 * random.random()))
                continue
            return response
        except (requests.Timeout, requests.ConnectionError) as e:
            failures.append(f"attempt {attempt}: {type(e).__name__}: {e}")
            if attempt < max_retries - 1:
                base = min(2 ** attempt * 2, 30)
                time.sleep(base * (0.8 + 0.4 * random.random()))

    raise MaxRetriesExceeded("All attempts failed:\n  " + "\n  ".join(failures))

There is an important difference: TransientError (which includes network hiccup and timeout) retries with the current session. BlockedError marks the session as flagged, which in turn triggers is_expired() and forces a new session creation.

Scaling concurrent sessions

At a small scale, the session pool model works just fine. However, if your scraper runs on 50+ machines simultaneously, you need all of them not to create new sessions each, but to stick to the one.

A Redis-backed session pool is a solution. Basically, Redis is a shared notepad that lives on a server and that every machine can read and write to, so that there is no chaos in sessions. Store each ProxySession object (serialized as JSON) with a TTL key:

import redis
import json

r = redis.Redis(host='localhost', port=6379, db=0)

def store_session(session: ProxySession, ttl: int = 600):
    key = f"session:{session.session_id}"
    r.setex(key, ttl, json.dumps(session.__dict__))

def load_session(session_id: str) -> Optional[ProxySession]:
    data = r.get(f"session:{session_id}")
    if data:
        return ProxySession(**json.loads(data))
    return None

If a proxy provider offers sticky sessions, your Redis only needs to track the session token, not the exit node state. A provider handling IP consistency and you dealing with cookies and headers is the preferred architecture for effective scaling because building and maintaining a distributed session pool is good 2-4 weeks of work. You need developers to do it - and add ongoing maintenance as websites update their detection systems.

Testing rotation layer

To be sure everything works fine, verify your rotation layer. Send 10 requests to see whether all of them really go via the same IP address.

import requests

def verify_session_stickiness(session_token: str, n: int = 10):
    proxy = f"http://user__sessid.{session_token}:pass@gw.dataimpulse.com:823"
    seen_ips = set()

    for i in range(n):
        r = requests.get("https://httpbin.org/ip", proxies={"https": proxy})
        ip = r.json()["origin"]
        seen_ips.add(ip)
        print(f"Request {i+1}: {ip}")

    print(f"\nUnique IPs: {len(seen_ips)}")
    assert len(seen_ips) == 1, f"FAIL: Session leaked across {len(seen_ips)} IPs"
    print("PASS: Session sticky across all requests")

verify_session_stickiness("your_session_token_here")

To verify headers, use https://httpbin.org/headers and compare headers across requests. Any differences in User-Agent, Accept-Language, or sec-ch-ua between requests of the same session are a problem.

If you want a deeper test, use bot.sannysoft.com or pixelscan.net. Those sites show what detection systems see - WebDriver flags, inconsistent browser APIs, JA3 anomalies, timezone/language mismatches. You see what exactly is wrong and at which layer. This is an easier, faster, and cheaper way than constant tries and errors.

Final thoughts

Scrapers that remain undetected are not the ones with the largest IP pools. They are the ones that mimic real users’ behavior the most persuasively - when the IP, the TLS fingerprint, the header profile, the cookie state, and the navigation flow all together create traffic patterns that are typical for a genuine human visitor.

IP rotation is necessary, but it is a tactic, while session architecture is a strategy. When you adopt that approach, you feel the difference - in lower block rates and higher data accuracy, in how much less firefighting your team goes through every time security systems update. Scalability is another benefit.

Also, there is an undeniable tendency for detection systems to develop, and so must your scrapers do. Rotation that is based on sessions, not single IPs, allows for that development.

If you want to skip the “build infrastructure from scratch” part, DataImpulse is an ethical proxy provider that offers both rotating and sticky sessions. You can set the duration of a session manually - up to 120 minutes is possible. DataImpulse (dataimpulse.com) provides 90M+ of residential, mobile, and datacenter enterprise-grade addresses in 195 locations with advanced targeting options (extra costs) to help maintain session coherence. Web scraping, SERP tracking, price monitoring, and other sensitive scenarios are among its use cases. Prices start from $1/GB - for non-expiring traffic and 24/7 human support.

Among the many tools a scraping engineer should master are headless browsers. But if you want to be completely honest, in the scraping industry there’s some confusion about the concepts of “headless browsers” and “browser automation tools”.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

---

In this article, you’ll get your ideas clear on headless browsers and browser automation tools. You’ll also discover the Oxylabs Headless Browser and learn how to use it in practice against common anti-bots. In particular, you’ll see the difference in servers’ responses when hitting the same target URLs with and without the Oxylabs solution.

Let’s get into it!

What is a Headless Browser?

A headless browser is a fully functional web browser that operates without a graphical user interface (GUI). To understand what this means in practice, it helps to discuss what a browser actually does under the hood.

When you open a browser like Chrome and navigate to a URL, the browser performs a precise sequence of operations:

• It sends an HTTP request to a server.

• It receives an HTML document.

• It parses that HTML into a DOM (Document Object Model) tree.

• It fetches and applies CSS stylesheets to compute the visual layout, downloads and executes JavaScript files, and processes any asynchronous network requests (XHR/Fetch API calls) triggered by that JavaScript.

• It renders everything into visible pixels on your screen.

Below is an image that graphically shows the whole process:

A headless browser performs every single one of those steps, except painting pixels to a physical display. It still builds the DOM, executes JavaScript, processes AJAX calls, and maintains a fully accurate representation of the page state in memory. But it does it without needing a graphical rendering environment.

It’s also important to understand the difference between a headless browser and a basic HTTP client. Tools like Python’s requests library or curl are pure HTTP clients. This means that they send a request and return whatever raw response the server provides. They have no concept of a DOM, no JavaScript engine, and no rendering pipeline.

The most widely used headless browsers are the following:

• PhantomJS: One of the first dedicated headless browsers, built on an older version of WebKit. It was widely used before 2017 but has since been officially abandoned and is no longer maintained. You may still encounter it in legacy codebases, but it’s better not to use it for new projects.

• Headless Chrome/Chromium: Google officially introduced headless mode in Chrome 59 (2017), effectively making PhantomJS obsolete overnight. It exposes the Chrome DevTools Protocol (CDP) as its programmatic interface, which automation frameworks connect to to send instructions and receive data.

• Headless Firefox: Mozilla introduced headless support in Firefox 55. Powered by the Gecko rendering engine and the SpiderMonkey JavaScript engine, it’s a strong alternative to Chromium, particularly useful when you need to scrape or test behavior tied to the Gecko engine specifically.

• WebKit: WebKit is the browser engine that powers Apple’s Safari. Playwright bundles a version of WebKit that can be run in headless mode on non-Apple platforms, giving you the ability to simulate Safari’s rendering behavior on Linux or Windows servers (something that would be otherwise impossible).

• Splash: A lightweight, scriptable headless browser built specifically for web scraping, developed by the Scrapy team. It is written in Python and Lua, exposes an HTTP API, and integrates natively with the Scrapy framework, making it a beginner-friendly option for Python-based scraping pipelines.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

Headless Browsers vs. Automation Frameworks: A Critical Distinction

At this point, it’s essential to draw a clear architectural boundary between two categories that are frequently, but incorrectly, used interchangeably: headless browsers and browser automation frameworks.

As said in the previous paragraph, a headless browser is the actual browser engine itself. So, it’s the software responsible for all the core browser operations described above.

A browser automation framework, on the other hand, is a programming library or tool that sits on top of a headless (or full) browser and provides a developer-friendly API to control it programmatically. The framework itself does not render pages, execute JavaScript, or manage network traffic: it delegates all of that to the underlying browser engine. What it provides is an abstraction layer to navigate to URLs, click elements, fill in forms, wait for conditions, intercept network requests, take screenshots, and to perform any similar activity.

Think of it this way: the headless browser is the engine of a car, and the automation framework is the steering wheel, pedals, and dashboard. You need both to drive, but they are fundamentally different components with different responsibilities.

This distinction also has a practical implication. If a website employs bot detection based on browser fingerprinting, it’s the underlying browser engine’s characteristics that matter, not the automation framework on top of it. This means that switching from Puppeteer to Playwright doesn’t change your browser fingerprint if both are driving the same Chromium engine with the same configuration.

The list below reports the programming libraries and tools used to control headless (or full) browsers programmatically:

• Puppeteer: A Node.js library developed and maintained by Google that provides a high-level API to control Headless Chrome/Chromium over CDP. It was the first major modern automation framework to replace PhantomJS and remains widely used in the JavaScript/Node.js ecosystem.

• Playwright: Developed by Microsoft, Playwright is a more modern and feature-rich automation framework that supports Chromium, Firefox, and WebKit under a single unified API. It has become the preferred choice for many scraping engineers due to its superior handling of async operations, automatic waiting strategies, built-in network interception, and robust multi-browser support. On the side of programming languages, it’s available for Node.js, Python, Java, and .NET.

• Selenium: The oldest and most widely adopted browser automation framework in existence, originally released in 2004. Unlike Puppeteer and Playwright, Selenium communicates with browsers via the W3C WebDriver protocol. This means it supports a broader range of browsers and is the de facto standard in enterprise testing environments. Selenium supports multiple languages, including Python, Java, C#, Ruby, and JavaScript, which gives it an exceptionally large community and an extensive ecosystem of documentation and tooling. For engineers working in Python, Selenium with a headless Chrome or Firefox driver is still a very common scraping approach. Its main trade-off is that it can be slightly slower and less ergonomic for modern async-heavy scraping scenarios.

Why Do Engineers Need a Headless Browser?

When you start defining how to scrape a website, your first instinct is often to reach for a lightweight HTTP client. And, in many cases, that is the right call. Simple, server-rendered websites that return complete HTML in their initial response are perfectly scrapable with tools like requests coupled with BeautifulSoup.

However, as you begin working with modern websites, you will quickly encounter scenarios where an HTTP client falls flat. Understanding why this happens (and when a headless browser becomes necessary) is one of the most fundamental skills you need to master for your career.

This section discusses the most common scenarios when you need to use a headless browser for scraping target websites.

JavaScript-Rendered and SPA Content

The most common reason to turn to headless browsers is client-side JavaScript rendering. A large proportion of modern websites, in fact, are built as Single Page Applications (SPAs) using frameworks like React, Vue.js, Angular, or Next.js. In these architectures, the server does not send a fully populated HTML page. Instead, it sends a minimal HTML shell along with a bundle of JavaScript files. The browser then executes that JavaScript, which makes additional API calls to fetch the actual data and dynamically constructs the DOM.

If you scrape the raw HTML with an HTTP client, you will receive that empty shell and find none of the content you are looking for. A headless browser, instead, loads the page as a real user’s browser would, waits for the JavaScript to finish executing, and exposes the fully populated DOM for you to extract data from.

AJAX and Asynchronous Data Loading

Closely related to SPAs is the broader pattern of asynchronous data fetching. Even on websites that are not full SPAs, it’s extremely common for certain page sections to be populated via AJAX requests that fire after the initial page load. These requests hit internal or external APIs, and the responses are injected into the page dynamically.

With a headless browser and a tool like Playwright, you can intercept these network requests directly. Also, this is a powerful web scraping technique. Rather than scraping the rendered HTML, you capture the raw API response (usually as JSON), which gives you clean and structured data to work with.

Infinite Scroll and Pagination

Many websites like social media platforms, job boards, and e-commerce product listings implement infinite scroll. This means that new content is loaded dynamically as the user scrolls down the page. On the practical side, this means that there is no “Page 2” URL to follow. The content doesn’t exist in the DOM until a scroll event triggers a new batch of data to load.

In this scenario, a headless browser allows you to simulate scroll events, wait for new content to be injected, and continue extracting data until a stopping condition is met. This is fundamentally impossible to replicate with a plain HTTP client.

User Interaction Simulation

Some content is only accessible after specific user interactions, like clicking on a button, selecting a value from a dropdown menu, submitting a search form, switching between tabs, and similar. A headless browser gives you a full interaction API that allows you to click on elements, type into input fields, select options, trigger keyboard shortcuts, and simulate virtually any user action.

This is particularly important when scraping search-driven websites like travel booking sites, real estate portals, and any similar websites where results are generated after the user performs actions.

Bot Detection and Browser Fingerprinting Evasion

Many high-value websites deploy anti-bot systems that analyze incoming requests to determine whether they originate from a real browser or an automated script. These systems are trained using machine learning models to check for dozens of signals. A headless browser provides a much more convincing browser environment than a raw HTTP client.

In particular, with additional configurations and stealth patches, you can significantly reduce the likelihood of your scraper being detected and blocked. Of course, this is an arms race: anti-bot technology continues to evolve, and a headless browser alone is not a silver bullet.

Check the TWSC YouTube Channel

What Is Oxylabs Headless Browser?

Oxylabs Headless Browser is a cloud-based browser automation solution that allows engineers to run and control remote browser sessions without the overhead of managing browser infrastructure locally or on their own servers. Rather than spinning up a local Chromium or Firefox instance on your own machine or cloud VM, Oxylabs Headless Browser offloads all of that complexity to dedicated remote servers maintained by Oxylabs.

The result is a production-ready and fully managed browsing environment that you connect to via a standard WebSocket (WSS) connection, and control using any automation framework that supports the Chrome DevTools Protocol (CDP) exactly as you would a local browser instance.

The image below shows, at a high level, the architectural difference between traditional browser automation and using Oxylabs’ solution:

The currently available browser instance is a Chrome-based environment (ubc.oxylabs.io). This runs high-performance Chromium instances with advanced stealth configurations on dedicated servers with integrated residential proxies.

Key Features

What differentiates Oxylabs Headless Browser from a self-managed setup is the set of capabilities that are baked directly into the infrastructure. Rather than requiring you to implement them yourself, it provides you with the following:

• Integrated residential proxies: Every browser session is automatically routed through Oxylabs’ residential proxy network. This eliminates the need to manage a separate proxy integration and ensures that outgoing requests carry legitimate residential IP addresses, reducing the risk of IP-level blocks.

• Automatic CAPTCHA solving: The platform detects and solves CAPTCHAs by default on page load. For CAPTCHAs that appear mid-session, it exposes an event-based API via the window object that your script can subscribe to in order to pause and resume automation around the solving process.

• Geolocation targeting: You can pin your browser session to a specific country, state, or city by appending query parameters directly to the WebSocket connection URL. This is essential for scraping geo-targeted content, verifying localized pricing, or accessing region-restricted pages without maintaining a geographically distributed proxy infrastructure yourself.

• Device type emulation: The p_device parameter allows you to emulate desktop, mobile, or tablet environments. This gives you access to device-specific content, responsive layouts, and mobile-only UI elements or CAPTCHA variants.

• Session inspection: This is a debugging feature that uses VNC (Virtual Network Computing) to give you real-time visual access to a running browser session. This is invaluable when diagnosing issues that are not apparent from logs alone. For example, for unexpected page rendering, broken interaction flows, or timing-related problems in your automation script.

• MCP integration: Oxylabs Headless Browser supports connection via the Model Context Protocol (MCP), enabling AI systems such as Claude Desktop or Cursor to drive browser automation sessions autonomously, which opens the door to AI-driven scraping and web navigation workflows.

How Oxylabs Headless Compares to Self-Managed Headless Browsers

The practical significance of Oxylabs Headless Browser is best understood by contrasting it with the self-managed alternative. When you run Playwright or Puppeteer against a local Chromium instance, you are responsible for the entire stack:

• Installing and maintaining the browser binary.

• Managing proxy rotation and authentication.

• Implementing CAPTCHA-solving integrations.

• Applying stealth patches to avoid fingerprinting.

• Handling concurrency limits.

• Ensuring your infrastructure scales with your scraping workload.

Oxylabs Headless Browser, instead, abstracts all of that away behind a single WSS endpoint. In practice, this means that your code remains identical to what you would write against a local browser. The connection string is the only thing that changes.

This makes Oxylabs’ solution particularly suited for teams that need to scale browser-based scraping quickly without investing in dedicated infrastructure engineering.

As a final note, it’s worth considering that billing is based on traffic consumed (per GB). This means traffic optimization becomes an important cost management practice when operating at scale.

Need help with your scraping project?

Testing Common Anti-bots The Hard Way: Step-by-step Tutorial on How to Hit Them Headless Mode

In this section, you will learn how to hit common anti-bots in headless mode using Playwright. The idea is to compare the results obtained in headless mode with the ones obtained using Oxylabs Headless Browser.

The target websites with their anti-bots are:

• Ssense → Cloudflare

• Gucci → Akamai

• Loccitane → DataDome

• Empire-cat→ Kasada

Let’s get into the tutorial!

Prerequisites and Requirements

To replicate this tutorial, you need at least Python 3.8 installed on your machine.

Create a new folder for your project:

mkdir test-antibots

At the end of this step, the test-antibots/ folder will have the following structure:

test-antibots/
   |
   |-- tester.py
   |
   |__ venv/

Where:

• tester.py is the Python file that contains the scraping logic to test the antibots.

• venv/ is the folder that contains the virtual environment.

All the files can also be downloaded from our GitHub repository, available to anyone.

In the activated virtual environment, install the needed libraries:

pip install playwright chromium

Very well. You are ready to code your tester!

Step #1: Code The Scraper to Test the Anti-bots

To test the antibots, write the following in tester.py:

from playwright.sync_api import sync_playwright

# Target pages with their anti-bots
targets = [
    ("<https://www.ssense.com/en-us/men/designers/acne-studios>", "Cloudflare"),
    ("<https://www.gucci.com/us/en/pr/women/shoes-for-women/sandals-for-women/flat-sandals-for-women/womens-slide-sandal-with-horsebit-p-834427AAE046273>", "Akamai"),
    ("<https://loccitane.com/en-us/olio-jeunesse-divine-30ml-27DH030I22.html>", "DataDome"),
    ("<https://www.empire-cat.com/equipment/used/catalog/aa/j1s03815>", "Kasada")
]

results = []

# List of titles or substrings that indicate common blocks/challenges
BLOCK_INDICATORS = [
    "just a moment", "access denied", "attention required", 
    "cloudflare", "ddos protection", "captcha", "verify you are human"
]

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context()

    for url, antibot in targets:
        page = context.new_page()
        try:
            # We capture the response to check the status code
            response = page.goto(url, wait_until="domcontentloaded")
            page.wait_for_timeout(5000)

            title = page.title() or ""
            status_code = response.status if response else 0
            
            # Validation Logic
            is_blocked = False
            error_reason = ""

            # 1. Check HTTP Status (403 Forbidden is a classic block)
            if status_code >= 400:
                is_blocked = True
                error_reason = f"HTTP {status_code}"
            
            # 2. Check for "Waiting Room" or "Block Page" titles
            elif any(indicator in title.lower() for indicator in BLOCK_INDICATORS):
                is_blocked = True
                error_reason = "Challenge/Block Page detected"
            
            # 3. Check for empty titles (Common with Kasada/DataDome blocks)
            elif not title or title.strip() == "":
                is_blocked = True
                error_reason = "Empty/N/A Title (Hidden Block)"

            if is_blocked:
                results.append({"url": url, "antibot": antibot, "title": title, "status": "fail", "error": error_reason})
                print(f"❌ [{antibot}] Blocked: {error_reason}")
            else:
                results.append({"url": url, "antibot": antibot, "title": title, "status": "ok"})
                print(f"✅ [{antibot}] Success: {title}")

        except Exception as e:
            error_msg = str(e).split('\n')[0] # Keep it brief
            print(f"❌ [{antibot}] Exception: {error_msg}")
            results.append({"url": url, "antibot": antibot, "title": None, "status": "fail", "error": error_msg})

        finally:
            page.close()

    browser.close()

# Print summary table
passed = sum(1 for r in results if r.get("status") == "ok")
failed = len(results) - passed

print("\n" + "=" * 105)
print(f"{'STATUS':<10} {'ANTIBOT':<12} {'RESULT/REASON':<30} {'URL'}")
print("=" * 105)
for r in results:
    status = "✅ PASS" if r.get("status") == "ok" else "❌ FAIL"
    antibot = r.get("antibot", "N/A")
    # Show the reason if it failed, otherwise show the page title
    display_info = r.get("error") if r.get("status") == "fail" else r.get("title", "N/A")
    print(f"{status:<10} {antibot:<12} {str(display_info)[:28]:<30} {r['url']}")
print("=" * 105)
print(f"✅ Passes: {passed}  |  ❌ Fails: {failed}  |  Total: {len(results)}")
print("=" * 105)

This snippet does the following:

• Maps a list of target URLs, each associated with their anti bot.

• Uses Playwright to launch a Chromium browser in “headless” mode.

• For each target URL, the script:

  • Opens a new browser page.

  • Attempts to navigate to the site and waits for the initial content to load.

  • Pauses for 5 seconds to see if the page successfully renders the correct content or if it gets blocked/redirected by the site’s security.

  • Attempts to extract the page title as proof of a successful load.

• Avoids the common trap of assuming a successful page load implies successful data access by checking the HTTP status code, title semantics, and structural integrity (line “N/A” titles)-

• Prints summarized results.

Perfect. The code is ready for launch.

Step #2: Launch The Tester And Retrieve Results

After launching the scraper, the results are the following:

STATUS     ANTIBOT      RESULT/REASON                  URL
==========================================================================================
❌ FAIL     Cloudflare   HTTP 403                       <https://www.ssense.com/en-us/>...
❌ FAIL     Akamai       Page.goto: net::ERR_HTTP2_PR   <https://www.gucci.com/us/en/pr/>...
❌ FAIL     DataDome     Empty/N/A Title (Hidden Bloc   <https://loccitane.com/en-us/olio->...
❌ FAIL     Kasada       HTTP 429                       <https://www.empire-cat.com/equi>...
===========================================================================================
✅ Passes: 0  |  ❌ Fails: 4  |  Total: 4
===========================================================================================

As you can see, the anti-bots successfully blocked 4 requests on 4:

• Cloudflare returned a 403 (forbidden).

• Akamai returns a Page.goto: net::ERR_HTTP2_PR error, which is often related to fingerprinting.

• DataDome reported an empty result, which suggests it may have activated a captcha.

• Kasada returns a 429 error. Generally speaking, a 429 error is a “too many requests” error, so the server blocks your scraper because you’re hitting it too fast. But specifically for Kasada, this often indicates a block due to fingerprinting.

Terrific! You made it to the end of this tutorial. Now it’s time to see what happens when implementing the Oxylabs solution.

Testing Common Anti-bots With Oxylabs Headless Browser: Step-by-step Tutorial

In this tutorial section, you will learn how to hit the same websites as in the previous paragraph using Oxylabs Headless Browser.

Prerequisites and requirements

Other than the requirements previously met, you need an active Oxylabs account in order to create your username and password for using its headless browser. Also, make sure to match all the requirements needed by the Oxylabs Headless Browser.

In the test-antibots/ folder, create a new Python file that will contain the scraping logic using the Oxylabs product:

test-antibots/
   |
   |-- tester.py
   |
   |-- tester-oxylabs.py
   |
   |__ venv/

Good. You are ready to proceed with the actual tutorial.

Step #1: Code The Scraper Using Oxylabs Headless Browser

Write the following code in tester-oxylabs.py:

import json
import random
import string
import os
from playwright.sync_api import sync_playwright

# Define params for connecting to Oxylabs servers
username = "<your-oxylabs-username>"
password = "<your-oxylabs-password>"
endpoint = "ubc.oxylabs.io"

# Create folder for screenshots
if not os.path.exists("screenshots"):
    os.makedirs("screenshots")

def get_session():
    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=10))

# Define Chrome arguments
chrome_args = json.dumps([
    "--disable-blink-features=AutomationControlled", 
    "--enable-features=NetworkServiceInProcess"
])

# Define params for Oxylabs Headless Browser
params = f"p_device=mobile&p_cc=US&p_session_id={get_session()}&args={chrome_args}"
browser_url = f"wss://{username}:{password}@{endpoint}?{params}"

targets = [
    ("<https://www.empire-cat.com/equipment/used/catalog/aa/j1s03815>", "Kasada"),
    ("<https://www.ssense.com/en-us/men/designers/acne-studios>", "Cloudflare"),
    ("<https://loccitane.com/en-us/olio-jeunesse-divine-30ml-27DH030I22.html>", "DataDome"),
    ("<https://www.gucci.com/us/en/pr/women/shoes-for-women/sandals-for-women/flat-sandals-for-women/womens-slide-sandal-with-horsebit-p-834427AAE046273>", "Akamai")
]

# Define semantics to intercept in responses
BLOCK_TITLES = ["just a moment", "access denied", "attention required", "security measure", "not acceptable", "403 forbidden"]
results = []

def run_benchmark():
    with sync_playwright() as p:
        for url, antibot in targets:
            print(f"---\nTesting {antibot}...")
            try:
                browser = p.chromium.connect_over_cdp(browser_url)
                context = browser.new_context()
                page = context.new_page()

                captcha_detected = False
                def handle_log(msg):
                    nonlocal captcha_detected
                    if "oxylabs-captcha-solve-start" in msg.text:
                        print(f"   [Solver] {antibot} challenge detected...")
                        captcha_detected = True
                    elif "oxylabs-captcha-solve-end" in msg.text:
                        print(f"   [Solver] {antibot} resolved!")
                        captcha_detected = False

                page.on("console", handle_log)
                page.goto(url, wait_until="domcontentloaded", timeout=60000)
                
                # Wait for the page to stabilize
                if captcha_detected:
                    page.wait_for_timeout(15000)
                else:
                    page.wait_for_timeout(8000)

                # Make and save screenshots
                screenshot_path = f"screenshots/{antibot.lower()}_result.png"
                page.screenshot(path=screenshot_path, full_page=False)
                print(f"   [Camera] Screenshot saved: {screenshot_path}")

                title = page.title() or ""
                low_title = title.lower()
                is_blocked = any(bt in low_title for bt in BLOCK_TITLES)
                
                if len(title) > 5 and not is_blocked:
                    print(f"✅ {antibot} PASS: {title[:30]}...")
                    results.append({"url": url, "antibot": antibot, "status": "ok", "title": title})
                else:
                    print(f"❌ {antibot} FAIL: {title}")
                    results.append({"url": url, "antibot": antibot, "status": "fail", "error": title})

                browser.close()

            except Exception as e:
                print(f"❌ {antibot} Exception: {str(e)[:50]}")

run_benchmark()

Below is what changes with respect to the version in the previous paragraph:

• It defines the credentials for connecting to Oxylabs servers and creates a folder where it will later store screenshots.

• With chrome_args(), it defines some parameters for setting the automation in Chrome.

• In the string params, the p_mobile and *p_cc* respectively manage the perceived device type and IP.

• It manages CAPTCHAs with oxylabs-captcha-solve-start and oxylabs-captcha-solve-end. And since it needs to manage CAPTCHAs at each request, the code now has a timeout that helps the pages stabilize before the scraper passes on to the next target URL.

• It uses page.screenshot() to make screenshots after the server sends the request and saves them into the previously created folder. This is important because the exceptions can’t all be handled via intercepting HTTP responses or via semantics. By making a screenshot before the connection is closed, you can actually see if the scraper has passed the anti-bot or if it failed at it.

Good. You’re now ready to run the code and see the results.

Step #2: Launch The Script

After launching the scraper, the results are the following:

---
Testing Kasada...
   [Camera] Screenshot saved: screenshots/kasada_result.png
✅ Kasada PASS: 2021 CATERPILLAR 950M | Empire...
---
Testing Cloudflare...
   [Camera] Screenshot saved: screenshots/cloudflare_result.png
✅ Cloudflare PASS: Latest Acne Studios Collection...
---
Testing DataDome...
   [Camera] Screenshot saved: screenshots/datadome_result.png
✅ DataDome PASS: Immortelle Anti-Aging Face Oil...
---
Testing Akamai...
   [Camera] Screenshot saved: screenshots/akamai_result.png
✅ Akamai PASS: Slides for Women | Designer Sl...

As you can see, using the Oxylabs Headless Browser, the scraper overcomes all the anti-bots!

As a final note, consider that, based on the configuration and settings you will actually implement, DataDome (but also other anti-bots) may activate sliding captchas:

In this case, the Oxylabs Headless Browser is of no help as it only supports hcaptcha, recaptcha, and turnstile dynamic captchas.

Hooray! You successfully made it to your first scraping project using the Oxylabs Headless Browser.

Conclusion

In this article, you learned the actual distinction between headless browsers and browser automation tools. You also discovered the Oxylabs Headless Browser which, despite the name, it’s cloud-based browser automation. The difference between this and the browser automation tools you already know is that it’s cloud-based. This means that when it’s called via the code, it manages all the infrastructure on its own without overhead on your side.

As you learned in the tutorial section, it’s also pretty easy to configure and to use: you just need to manage some parameters and call the Oxylabs endpoint.

So, let us know: were you already using the Oxylabs Headless Browser?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

---

The first is the UDP we want, HTTP/3 for our requests. The second is the UDP we never asked for, the WebRTC traffic that a browser generates on its own, which can leak around the proxy and expose our real IP. But supporting UDP and actually getting a client to route its UDP through the proxy are two different problems, and the result changes a lot depending on the client. An HTTP client, a browser doing HTTP/3, and a browser doing WebRTC all behave differently, so we tested all three, end to end, to see where a UDP proxy does the work for us and where we still have to step in on the client side.

A quick recap of HTTP/2 and HTTP/3

We already covered the request side of HTTP/3 in Faster Web Scraping with HTTP/3, so let’s just recap what matters for proxies. HTTP/1.1 and HTTP/2 run on TCP. HTTP/3 drops TCP completely and runs on QUIC, which is built on UDP and merges the transport and TLS handshakes into one. The part that matters for us is the transport: a proxy built to relay TCP has nothing to grab onto in a QUIC connection, since there’s no TCP stream to relay.

Adoption is real, but it’s easy to overstate. If we look at traffic, HTTP/3 is around a fifth of all requests, while HTTP/2 is still the majority at almost half, and this split has stayed roughly the same through 2026 (Cloudflare Radar). If we look at site support instead, the number goes up: around 38% of sites advertise HTTP/3 via Alt-Svc (W3Techs). Italy, where our test machine sits, is one of the higher HTTP/3 markets at about 30%. So HTTP/3 is a serious, well-supported protocol, not some niche thing, but it hasn’t replaced HTTP/2 yet, and most real traffic is still on TCP.

HTTP/3 and QUIC run on UDP, so the proxy layer has to carry it too. anyIP gives compatible clients production-ready SOCKS5 UDP ASSOCIATE across its residential and mobile network.

Explore anyIP UDP Proxies

The sales pitch follows naturally from here. If real browsers increasingly speak HTTP/3 and our scraper can’t, that difference becomes a signal, so we’d want to speak HTTP/3 too, and to do that through a proxy, the proxy has to carry UDP.

This is exactly what some providers are now selling. They offer UDP support, usually as a beta over SOCKS5, and they pitch it as a stealth feature: the idea is that most residential traffic is QUIC these days, so a client stuck on TCP stands out. There are two parts of this claim worth checking before relying on it, the stealth side and the tooling side, so we tested both, using a provider that carries UDP and one that doesn’t.

UDP support and the stealth question

Switching to HTTP/3 doesn’t remove protocol fingerprinting, it just moves it somewhere else. Detection happens at three layers, and moving from TCP to QUIC affects each one in a different way.

The TCP/IP stack fingerprint, meaning the SYN options, window sizes and TTL that give away the OS, really does disappear, because there’s no TCP handshake anymore. That’s the only layer that goes away, though. The TLS fingerprint stays exactly as it was, because QUIC still does a TLS 1.3 handshake, just wrapped inside QUIC instead of running over TCP. The ClientHello with its cipher suites and extensions is still there, and JA4 was designed to fingerprint the QUIC variant on purpose (Scrapfly QUIC/HTTP3 fingerprint). The HTTP/2 fingerprint, the SETTINGS frames, window updates and pseudo-header order, doesn’t get erased either. It just gets replaced by an HTTP/3 fingerprint built from QUIC transport parameters.

And those QUIC transport parameters are basically the same thing we were trying to get away from. Scrapfly says it clearly: they “work like HTTP/2 SETTINGS frames but at the transport layer.” Values like max_udp_payload_size and initial_max_data, and the order they come in, are different between Chrome’s quiche, Firefox, Go’s quic-go and ngtcp2, and that difference is itself a fingerprint (Scrapfly HTTP/2 and HTTP/3 fingerprinting). QUIC even adds new signals that have no TCP equivalent at all, like 0-RTT behavior and connection migration support.

So the layer that actually gets scrapers blocked today is TLS and HTTP/2 fingerprinting, which Cloudflare, Akamai and DataDome score together as JA4 and JA4H. That’s a TCP-side problem, and it’s one we already solve with a client that imitates a browser’s handshake. HTTP/3 fingerprinting is growing but still immature. Scrapfly is pretty direct about it: it “adds signals on top of” HTTP/2 instead of replacing it, real Chrome on HTTP/2 is still accepted on the vast majority of sites, and the scraper-on-HTTP/2 versus user-on-HTTP/3 gap is only “starting to become a signal on Cloudflare’s premium tier.” There’s also a catch worth keeping in mind: a scraper that speaks HTTP/3 through a non-browser QUIC stack shows a QUIC fingerprint that matches no real browser, so it can stand out just as easily as it blends in. In other words, a UDP proxy is what gets us onto HTTP/3 in the first place, but the stealth benefit only pays off if the client presents a QUIC fingerprint that a real browser would. The proxy just carries the traffic; making it look right is still our job.

Check the TWSC YouTube Channel

The tooling has not caught up yet

The other half of the pitch is the tooling. The most common client for browserless scraping is curl_cffi, so the real question is whether it can actually do HTTP/3 in the setup a scraper normally runs in. We tested it (run_curl_cffi_http3.py), and you can find the code, as always, on our GitHub repository available for our paid readers, under the folder 109.UDP

The library is compiled with the QUIC stack, ngtcp2 and nghttp3, and on a plain direct request it does reach HTTP/3. It even reaches HTTP/3 with a full Chrome fingerprint if we use the low-level interface, so impersonation and HTTP/3 aren’t really in conflict, even if the first error makes it look that way.

high-level requests API:
  plain, no impersonation, V3ONLY        -> HTTP/3
  Chrome impersonation, V3ONLY           -> ERROR: QUIC needs at least TLS version 1.3
  Chrome impersonation, HTTP proxy, V3   -> ERROR: HTTP/3 is not supported over an HTTP proxy
  Chrome impersonation, SOCKS5, V3       -> ERROR: HTTP/3 is not supported over a SOCKS proxy
low-level Curl API (same impersonation):
  impersonate chrome + V3ONLY            -> HTTP/3

Need help with your scraping project?

That middle error in the high-level API is just a TLS-version conflict inside the requests wrapper, not a real limitation, since the low-level call combines the Chrome fingerprint with HTTP/3 and gets there fine. The lines that really matter are the proxy ones. curl refuses HTTP/3 over a proxy completely, both HTTP and SOCKS, and it refuses before it even connects, so this is curl saying no, not the proxy. It happens even with a proxy that could carry the UDP. And since a scraper always runs through a proxy, in practice curl_cffi ends up being HTTP/2 over TCP, and a UDP-capable proxy doesn’t change that at all.

So this is the honest picture on the request side. A proxy that carries UDP does its job for a client that tunnels its own UDP, but the popular HTTP clients refuse HTTP/3 over a proxy right now, so the benefit is stuck waiting for the tooling to catch up. The proxy capability is arriving before the clients that could use it, which is good to know before expecting HTTP/3 to flow through a scraper just because the proxy supports UDP.

The WebRTC leak, where UDP really bites

Read more

However, web scraping can also serve as a foundation for building the data infrastructure needed to better understand national markets and societal conditions.

In this blog post, I’ll show real-world examples of how web scraping is already used in the public sector, and how institutions could leverage it to improve policies and deliver higher-quality services to citizens.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

---

Why Web Scraping Matters for Public Institutions

Public institutions have access to statistical datasets from official sources. These are surely relevant, but they tend to provide a picture of the market only after events have already happened, sometimes months or even a year later.

That delay creates a challenge. After all, institutions need timely information to understand citizens’ current needs and respond effectively. Another issue is that relevant data is often scattered across many sources, such as different websites, web portals, and marketplaces.

This is where web scraping can help! By collecting large volumes of publicly available data from multiple sites, institutions can complement and enrich official statistics.

Data collection is only the first step, though. The real value comes from building a complete data pipeline that includes cleaning, deduplication, aggregation, geolocation, statistical modeling, and interpretation.

At Prague Crawl 2026, Bayernheimerová Klára presented a compelling example of this approach. Her talk showed how the Czech Ministry of Finance relies on rental listing data collected by an external provider from several real estate portals and processes it through an end-to-end pipeline based on a dedicated statistical methodology.

As I’ll present shortly, the result is a reliable source of insights that supports housing policy decisions and affordable housing programs across the country, while also providing a valuable service to citizens.

What’s important to understand is that the goal isn’t simply to collect raw data, but to power services such as interactive maps, calculators, and dashboards that are useful to both public institutions and citizens.

Case Study: Czech Rental Market Intelligence System

To see an actual application of web scraping in the public sector, I’ll now present the initiative developed by the Czech Ministry of Finance for rental price analysis across Czechia.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The Motivation Behind the Project

Just like in many other European countries, housing affordability has become a major social and economic challenge in the Czech Republic. Housing supply has struggled to keep up with demand, causing both property prices and rents to rise steadily across the country.

To confront this issue, the Czech Ministry of Finance needed detailed and up-to-date information on local rental markets. The initiative was launched specifically to fill that gap, creating a web data analysis process that gives a continuously updated view of rental prices across the entire country.

The objective is to provide a reliable foundation for housing policy, affordable housing programs, and public-facing tools that help citizens better resonate with local rental markets.

Web Data Sourcing

The data for this initiative has been collected using Apify, one of the largest marketplaces of ready-made tools for web scraping, automation, and AI.

If you aren’t familiar with Apify, it provides over 41,000 ready-made serverless cloud programs (called Actors) to automate a wide range of tasks, including web scraping across thousands of different domains.

These Actors are developed and maintained by the community (and in some cases by Apify itself) and run on Apify’s cloud infrastructure. You can use them directly through Apify Console via a no-code interface, or call them programmatically via API. They can also be integrated into workflows such as n8n, Make, Zapier, or AI agents via MCP.

Two main reasons influenced the choice of Apify:

1. It’s a Czech-based company.

2. It enables the Ministry of Finance to gather large-scale, up-to-date information on rental listings without maintaining scraping infrastructure in-house.

In general, a solution like Apify lowers the barrier for public-sector teams, as these may have limited engineering capacity.

Downstream Data Pipeline

At a high level, this is the downstream data pipeline implemented by the Czech Ministry of Finance:

1. Data ingestion: Collects rental listings via Apify-based web scraping from multiple Czech real estate portals.

2. Data cleaning: Sorts data and removes duplicates, incorrect entries, and inconsistencies to ensure it is up to date, accurate, and free from deviations or distortions, following the procedure defined in Decree No. 456/2024 Coll.

3. Geolocation and aggregation: Standardizes addresses, assigns cadastral units, and enriches listings with attributes such as size, amenities, and building type.

4. Statistical modelling: Uses hedonic regression with spatial and temporal weighting to estimate underlying rental price levels.

5. Output generation: Presents results as interactive price maps and calculators.

Note: The pipeline also includes data update and recycling via a rolling window approach. This process adds new listings while retiring outdated observations to keep the dataset current.

To make rental prices comparable across different locations, the methodology is based on a reference apartment. That represents a standardized apartment profile with predefined characteristics, such as size, type, furnishing level, etc.

This approach reduces differences caused by individual property features and enables more consistent comparisons of rental price levels across municipalities and cadastral areas.

Produced Output and Tools

The Czech Rental Housing Price Map consists of two main solutions:

• Interactive rent price map: Provides estimated rental prices at the level of municipalities and cadastral areas across the Czech Republic.

• Market rent calculator: A practical tool that calculates a statistical estimate of rental price levels based on a standardized apartment profile and selected property characteristics.

Important: The two solutions aren’t intended to determine the market rent of a specific apartment. Instead, they return statistical estimates based on a standardized reference apartment. So, they’re intended primarily for market monitoring, regional comparisons, and housing policy analysis.

Interactive rent price map

The interactive rent price map shows minimum, maximum, and median rents for each area based on market listings, adjusted per square meter for a standard unfurnished reference apartment. It covers four size categories from 1+kk/1+1 to 4+kk/4+1.

Note that the map is interactive, and you can zoom in and out to explore individual cadastral areas. It’s also updated four times per year to ensure current data.

Market rent calculator

The market rent calculator is available as a form on the Czech government website. It works as follows:

1. Select the territorial unit where you want to estimate the rent.

2. Choose the size category of the apartment.

3. Enter the floor area of the apartment (if you don’t know it, the calculator will automatically use a default value for the selected location).

4. Indicate whether the apartment is in a new building.

5. Optionally specify whether the building uses non-standard construction materials (e.g., other than brick or panel).

6. Optionally select additional features such as a terrace, furniture, or an assigned parking/garage space.

7. Click “Calculate rent” to obtain an estimate of the monthly market rent (CZK) and the corresponding price level, which represents the final estimated rent for the selected apartment profile.

Impact

The project provides a systematic and regularly updated overview of market rental prices across the entire Czech Republic, supporting consistent monitoring of housing market developments over time and across regions.

Beyond market observation, the outputs serve as a key evidence base for housing policy design and housing programmes within public administration. For example, the State Investment Support Fund (SFPI) relies on these results when working with its affordable rental housing schemes.

Web Scraping in the Public Sector: Extending the Model

The project carried out by the Czech Ministry of Finance is just one example of a much broader pattern: how web scraping can be used as a foundation for modern public-sector data systems.

In particular, the same pipeline logic can be applied across many other domains. For example, in labor markets, web scraping can be used to collect job postings, salary ranges, and skill requirements from recruitment platforms. This can help governments identify regional skill shortages and design more targeted education or reskilling programmes.

Similarly, in consumer price monitoring, scraping can track grocery prices, housing costs, and essential goods across regions, enabling better inflation tracking and cost-of-living analysis. Other potential public domains that can benefit from this include energy consumption and transport accessibility.

When combined, those datasets become even more powerful. For instance, housing data, job market data, and grocery price data could be combined into a broader “liveability” index. That would help citizens and policymakers assess the overall affordability of living in a given region.

Check the TWSC YouTube Channel

Building a Job Market Choropleth for Czech Districts

In this section, I’ll guide you through building a job market choropleth across Czech districts. The idea is to show how web scraping, combined with a complete data pipeline, can be utilized for other scenarios beyond the previous example.

Important: The project below isn’t related to the initiative from the Czech Ministry of Finance and has been created purely for illustrative purposes.

Prerequisites

To follow this tutorial section, make sure you have:

• An Apify account (a free plan is sufficient).

• Python 3.11+ installed locally.

I’ll also assume you have a Python project set up locally with the following libraries installed:

pip install pandas geopandas geopy matplotlib

These are the required dependencies for this project and will be used as follows:

• pandas: Load, clean, and manipulate the scraped job posting data before geospatial analysis.

• geopandas: Handle geographic data, perform spatial joins, and create district-level maps.

• geopy: Convert job location addresses into geographic coordinates through geocoding.

• matplotlib: Visualize the results by generating a choropleth map showing job density by district.

Step #1: Access the Indeed Jobs Scraper Apify Actor

Just like in the Czech Ministry of Finance example, you can use Apify to collect the source data. This saves you from building and maintaining a complete job scraping pipeline from scratch.

For this project, we’ll use Indeed as the data source. Indeed has a Czech version, and most listings include the full office address where employees are expected to work.

In detail, I recommend using the Indeed Jobs Scraper [PPR] Actor. This automates the extraction of job titles, salaries, locations, company information, and job descriptions from Indeed.

To get started, log in to your Apify account and select the “Apify Store” option from the left-hand navigation menu. Then, search for “Indeed Jobs Scraper [PPR]”:

Click on the Actor card to reach its page. Great!

Step #2: Run the Job Scraping Task

On the Actor page, you’ll find an input form that lets you configure the scraper before running it in the cloud. Select “Czech Republic” as the “Country” and enter “Software engineer” in the “Query” field:

Note: In a real-world scenario, configure the Actor’s inputs according to your needs.

Next, toggle the option to avoid duplicates so that the Actor takes care of data deduplication for you:

Click the “Save & Start” button to launch the scraping task. The Actor will start running directly in Apify Console. As the scraper progresses, you’ll see the extracted job postings appear in real time. Be patient, as the process may take a few minutes depending on the number of matching listings.

Step #3: Explore the Output and Export the Results

Once the run completes, you’ll be able to explore the scraped dataset directly in Apify Console. As you can see, the Actor returns structured Indeed job data:

In this case, the Actor retrieved 63 job postings. That may seem like a small number, but keep in mind that the scraper targets only jobs published during the last 14 days. This helps ensure that the dataset reflects the current state of the job market.

Next, switch to the JSON view and select “All fields”. You’ll notice that each job posting includes a location object containing address information for the position:

That location data is exactly what you need to visualize job openings on a map and analyze their geographic distribution across Czech districts.

Finally, open the “Storage” tab, select the “JSON” export format, and click “Download” to export the scraped dataset:

A file with a name similar to dataset_indeed-scraper_2026-06-18_11-54-00-555.json will be downloaded. Rename it to jobs.json and place it in your Python project’s root directory.

Step #4: Download the Required Czech GeoJSON Data

To visualize the distribution of job openings across Czech districts, you’ll need a GeoJSON dataset containing the geographic boundaries of those districts.

Generally, that type of data is open and publicly available. For example, one possible source is the siwekm/czech-geojson repository, which provides GeoJSON files for various Czech administrative divisions:

Download the okresy.json file, which contains the geographic boundaries of Czech districts (okresy in Czech).

Note: Although the okresy administrative system was officially abolished in 2003, these districts are still used for statistical analysis. This makes them a good choice for visualizing the distribution of job opportunities across the country.

Once downloaded, add the okresy.json file to your Python project. At this point, your project structure should look similar to this:

├── jobs.json
├── okresy.json
└── main.py

Step #5: Visualize the Job Data on a Map

In your Python file, define the complete pipeline with the following code:

# pip install pandas geopandas geopy matplotlib

import json
from geopy.geocoders import Nominatim
from geopy.extra.rate_limiter import RateLimiter
import pandas as pd
import geopandas as gpd
import matplotlib.pyplot as plt

# Load the scraped Indeed job openings
with open("jobs.json", "r", encoding="utf-8") as f:
    jobs = json.load(f)

# Load the list of addresses from the job postings
addresses = []
for job in jobs:
    loc = job.get("location", {})

    # Prefer fullAddress, fallback to formattedAddressLong or formattedAddressShort
    address = (
        loc.get("fullAddress")
        or loc.get("formattedAddressLong")
        or loc.get("formattedAddressShort")
    )

    # Filter out full-remote jobs with no address
    if address and address.strip().lower() not in ["home office", "remote"]:
        addresses.append(address)

addresses = list(addresses)

# Initialize geocoder for Czech job locations
geolocator = Nominatim(user_agent="cz_job_map")
geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1)

# Geocode all job addresses into latitude/longitude points
rows = []
for address in addresses:
    loc = geocode(f"{address}, Czech Republic")

    # Keep only successfully geocoded results
    if loc:
        rows.append({
            "address": address,
            "lat": loc.latitude,
            "lon": loc.longitude
        })

# Convert geocoded results into a DataFrame
df = pd.DataFrame(rows)

# Convert DataFrame into GeoDataFrame with point geometries
points = gpd.GeoDataFrame(
    df,
    geometry=gpd.points_from_xy(df.lon, df.lat),
    crs="EPSG:4326"
)

# Load Czech administrative districts (okresy) from a local GeoJSON file
okresy = gpd.read_file("okresy.json")

# Align coordinate reference systems between datasets
if okresy.crs != points.crs:
    okresy = okresy.to_crs(points.crs)

# Assign each job point to its corresponding district
joined = gpd.sjoin(
    points,
    okresy,
    predicate="within",
    how="left"
)

# Count number of jobs per district
counts = (
    joined["id"]
    .value_counts()
    .reset_index()
)

counts.columns = ["id", "count"]


# Merge job counts back into district geometries
okresy = okresy.merge(
    counts,
    on="id",
    how="left"
)

# Replace missing values with 0 (districts with no jobs)
okresy["count"] = okresy["count"].fillna(0)

# Plot choropleth map with job counts per district
fig, ax = plt.subplots(figsize=(10, 8))

okresy.plot(
    column="count",
    cmap="Reds",
    edgecolor="black",
    linewidth=0.3,
    legend=True,
    vmin=0,
    ax=ax
)

# Add labels for districts with job activity
top = okresy[okresy["count"] > 0].sort_values("count", ascending=False)

for idx, row in top.iterrows():
    centroid = row.geometry.centroid

    ax.text(
        centroid.x,
        centroid.y,
        str(int(row["count"])),
        fontsize=9,
        ha="center",
        va="center",
        color=(lambda v: "white" if v > okresy["count"].max() * 0.5 else "black")(row["count"]),
        fontweight="bold"
    )

ax.set_title(
    "Software engineering job density on Indeed in the Czech Republic (last 14 days, by district)"
)
ax.axis("off")

plt.show() 

Need help with your scraping project?

That’s what the above script does:

1. Loads scraped job data from jobs.json and extracts valid physical addresses, filtering out remote or missing location entries.

2. Initializes the Nominatim geocoder with a rate limiter to safely convert addresses into latitude and longitude without exceeding request limits.

3. Geocodes each address into coordinates and keeps only successful results, storing them in a structured pandas DataFrame.

4. Converts the DataFrame into a GeoDataFrame with point geometries so the job data can be used in spatial analysis workflows.

5. Loads Czech district boundaries from the okresy.json file and ensures both datasets use the same coordinate reference system.

6. Performs a spatial join to assign each job point to a district, then aggregates job counts per district and merges results back into the map.

7. Visualizes the final dataset as a choropleth map, coloring districts by job density and adding readable labels for high-activity areas.

Step #6: Run the Script

Execute the script, and you’ll get a result like this:

Notice how most fresh software engineering openings are concentrated in districts surrounding Prague, Brno, and Ostrava. These are the three largest cities in the Czech Republic, so the result clearly makes sense!

Now, this was just a simple example, but you can use the same approach to build an interactive map or add additional features, such as calculating the average salary for each position by district, and more advanced analytics.

Value and Playbook for Other Governments

The approach presented in this blog post is highly adaptable to any public dataset with a spatial dimension, where location plays a key role in understanding patterns and inequalities. Still, a reusable country-independent playbook emerges:

1. Collect data via web scraping (or other administrative, public sources).

2. Clean and validate the data.

3. Enrich and aggregate it with contextual and geographic features.

4. Apply statistical or analytical models.

5. Present the results through maps, dashboards, or other interactive tools.

As you can tell, this pipeline isn’t domain-specific and can be reused across many different policy areas. Finally, it’s crucial that the adopted methodology remains open and clearly explained to the public. This fosters transparency, reproducibility, and trust in data-driven decision-making.

Conclusion

Here, I’ve shown how the Czech Ministry of Finance uses a web scraping service as part of a full data pipeline to turn rental listings into actionable insights for housing policy and public tools.

By accessing data collected from multiple real estate portals and processing it through cleaning, geolocation, and statistical modelling, they transform raw web data into interactive maps and rent calculators that support decision-making.

In this article, I built a similar pipeline for a different use case: a job market choropleth across Czech districts. I started from scraped Indeed listings, geocoded job locations, mapped them to districts, and visualized the results on a map.

I hope this example was useful and inspiring. If you have questions or ideas, feel free to share them in the comments below.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

You can read that episode two ways. Either the guardrails were the only thing standing between a public model and a stream of attack-ready output, or the shutdown was a tidier story than the truth. We are not going to settle that here. For our corner of the world, the lesson is simpler. A capable model, stripped of guardrails or jailbroken around them, automates the expensive part of an attacker’s job. Deobfuscation, code analysis, and figuring out how an unfamiliar system behaves under pressure. Those tasks have always gated the scraper-versus-vendor arms race because they cost human reverse-engineering hours, and that gate is now moving.

The cat-and-mouse game we cover every week runs on a hidden economy. A vendor ships an obfuscated sensor. Someone on the scraping side spends days or weeks reading minified JavaScript, renaming variables, and rebuilding the logic until the parameters make sense. The cost of that labor is what keeps most protections standing. If a model can do a real share of it on its own, the math behind the whole arms race changes.

Manus AI, and why not Claude

Manus is not a chat assistant. It is an agent that pairs a language model with a sandboxed virtual machine, and inside that machine, it runs code, fetches pages, beautifies scripts, and iterates on its own output. Meta acquired it at the end of December 2025 for around two billion dollars, though by June 2026, Chinese regulators were already forcing parts of that deal to unwind. What that architecture buys is verification. Manus not only describes what a script might do. It runs the script and checks.

That difference shows up the moment you set guardrails next to it. Ask Claude or ChatGPT to deobfuscate a live anti-bot sensor and explain which signals feed its scoring, and you tend to hit a wall. Those products keep tight cybersecurity guardrails, and the Fable shutdown is the clearest sign yet of how hard that line is enforced. We handed the same class of task to Manus, and it got to work. We have looked at agentic browsers and AI scraping assistants before, but those were aids for a human operator who still did the thinking. Here, the model does the reverse engineering.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Mapping Akamai on Net-a-Porter

We have written about Akamai more than any other anti-bot system. For the bypass-oriented background, THE LAB #30, THE LAB #85, and THE LAB #100 cover the recovery-when-nothing-works case, proxy chaining, and the browser-login-plus-HTTP hybrid. We will not repeat any of that.

What we wanted this time was not a bypass. It was a map. Akamai Bot Manager protects net-a-porter.com, and it defends in stacked layers that each fail differently:

• A TLS fingerprint check at the edge, evaluated before a single line of the client JavaScript runs.

• A sec-cpt cryptographic challenge, whose state lives in the bm_sc cookie.

• A behavioral sensor that serializes a binary telemetry payload and posts it to a collect endpoint, validating the _abck cookie.

• A client-side anti-hook layer that checks the browser environment for tampering and reports integrity separately.

The model we asked Manus to build covers the client sensor and the challenge flow, plus whatever it could observe about the TLS gate from the outside. A working end-to-end bypass was out of scope. We are not shipping code with this article, and we did not ask Manus to produce a runnable solver, even though our tests suggest we are really close to a working solution. What we’ll share here is a summary of the analysis Manus did; in the code repository, you will find the full details.

Check the TWSC YouTube Channel

What we handed the agent

The task was deliberately cold. We did not give Manus our own Akamai notes, our wiki, or any of the prior Lab articles. We pointed it at net-a-porter.com and asked it to study how Akamai's protection works, deobfuscate the client script, and explain which parameters feed into the sensor. Nothing more.

We chose this target for three reasons. It runs a real, current Akamai deployment on a high-value luxury store. The sensor was observable live, so the agent could fetch and instrument the actual script rather than reason about a stale sample. And we already understand Akamai well enough internally to judge whether the output was accurate or confident nonsense.

The reason the task went to Manus rather than a chat assistant is the guardrail gap from the previous section. Deobfuscating a production anti-bot sensor and enumerating the signals it scores is exactly the kind of request the heavily guarded models decline. Manus took it, ran the analysis in its VM, and came back with the two documents below.

What came back

The output was more detailed than we expected from a single task run. We are sharing it in two parts and deliberately not sharing it all.

The two documents below are Manus AI’s output. We reproduce them as reportage, to show what an agent derived on its own. We have not independently re-run every claim, re-derived every value, or confirmed that each detail still holds against the live site. Treat the specifics as the agent’s findings, not as TWSC-verified ground truth. The sensor version Manus reports, 746702557, was compiled on 2025-05-09, so some details may already have rotated. Where the agent surfaced operational specifics that amount to a turnkey recipe (exact module filenames, the full endpoint table, the CSS selector ids, the challenge request names), we have redacted them.

This is a schema of how Akamai Bot Protection according to Manus AI

Part one, reproduced in full, the anti-hooking and TLS summary

This first document is the safe exhibit. It explains how Akamai detects a tampered browser and how the TLS gate at the edge decides who even gets a challenge. We reproduce it verbatim.

<iframe height="0" width="0" style="display: none; visibility: hidden;" sc="true" tabindex="-1"></iframe>

Part two, reproduced with redactions, the deep technical analysis

The second document goes much further. It walks the bootstrapping sequence, the module system, the fingerprinting surface, and the telemetry transport. This is the part that starts to read like a blueprint, so we keep the architecture and the reasoning and we strip the operational specifics. Redaction markers below are ours, not Manus’s.

If you want to see the full analysis, you can find it in our GitHub repository reserved for paying users, inside the folder 108.AKAMAI-MANUS.

Read more

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

IP sourcing in the proxy industry

Two articles written by Spur Intelligence caught my attention in the past week.
It started with a webinar I’ve attended, where they explained what ISP proxies are and how proxy companies get those IPs.

The analysis is great, and I’ve understood better how the proxy industry works.
They also mentioned how NetNut gets high-quality ISP IP addresses, and described that in detail in this article.

How Proxy Providers Co-opt Entire Networks

Spur reverse-engineers how Netnut sources its ISP proxies through a partner called DiviNetworks, which installs GRE tunnels and policy-based routing directly on the border routers of real ISPs. The result is genuine ISP IP space sold as proxy inventory, with actual subscribers still living on the same addresses. The piece is worth reading for the detection detail: outbound connections sit in the TCP source port range 40,000 to 40,200, partner routers expose an rtr-<isp>.divinetworks.com passive-DNS pattern, and the Howard University case shows an entire /16 (AS919, around 17,000 exit IPs) co-opted at the network edge. They even quote DiviNetworks’ own figure of $13,208 a month for a US /16.
Full article here

Smart TV Apps and Residential Proxy SDKs

A more recent article, instead, highlighted how proxy companies get residential proxies. No surprises, at least on my side, in reading that Smart TVs are used as exit nodes for proxies. In fact, just like with mobile apps, developers can use SDKs from proxy providers to monetize their app installations. In some cases, it’s quite clear from the consent screen what’s happening under the hood.

All the industries that rely on web-scraped data (AI in primis, but not only) need residential proxies, so proxy providers, who are the gatekeepers for these tools, should be very careful about onboarding users with legitimate use cases and keeping fraudsters away. In my career, I’ve worked with almost every big name out there, and I’ve found this is the case. As scraping professionals, we should always remember that we’re guests (sometimes unwanted) on both the target website and the proxy infrastructure, so we should be as respectful as possible and not be driven by greed to collect data.
Full article here

Scraping Infrastructure

A great article by the engineering team at Browser Use shows how they’re spawning browsers cheaply and quickly.

How We Made Cloud Browsers 3x Cheaper and Faster

Browser Use walks through rebuilding its cloud so every session is its own Firecracker microVM, and the interesting twist is that they run Firecracker on plain EC2 instead of bare metal, accepting nested virtualization to get faster scale-up and lower cost.

The numbers are great: $0.02 per browser hour down from $0.06, VM cold start under 400ms, and create latency of 825ms at p50 across a 10,000-session test.

The engineering details that are making the difference are multiple, from mapping memory in 2MB pages plus a userfaultfd handler that preloads hot pages (resume-to-ready drops from 9.8s to 3.1s, roughly 91x fewer page-fault stalls) to two-phase vCPU pinning that took a 1,000-browser launch from 17% failed sessions to zero. They also make the case for running fully headless rather than headful, since their low-level Chromium fork pushes block-avoidance to 81% on their own benchmark.

Full article here

Read and share more of this

Want to flag and discover more articles like the ones above? Scraping News (still in beta) is where the community surfaces them. Sign up and submit what you find worth reading.

Want a concept map of everything covered here, cross-referenced with our own work and other sources worth your time?

The Web Scraping Club Wiki lives on GitHub, and as an interactive site. It is plain Markdown, so you can also clone it into your own Obsidian vault and read it locally.

You set it up, it works, and you forget about it. Three months later, someone asks: “Hey, did our competitor change their pricing last week?”

You check the pipeline dashboard, and everything looks fine: all green, no errors, last run successful some hours ago. Then you dig deeper and realize your scraper has been returning empty strings for weeks. The competitor did change their pricing, and you don’t know when this happened or what was on their website before.

In this article, you’ll learn how to build competitive scraping pipelines that make failure structurally (almost) impossible. Let’s get into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Why Competitive Intelligence Scrapers Break

Let’s start by understanding the most common failure modes that competitive scraping pipelines have:

• Structural drift: This is probably the most common. A competitor renames a CSS class, and your scraper starts returning empty strings. Your database fills with nulls, and nobody notices for weeks because the pipeline is still running.

• Soft blocks: These are nastier. You get valid HTTP responses, but the content is a CAPTCHA page or a bot-detection redirect. Your parser sees it as valid HTML, but it finds no data. So, it stores nothing, or, worse, it stores the CAPTCHA page’s text as if it were real data.

• Schema rot: This happens when a competitor evolves their product, but your data model doesn’t. They add a new pricing tier or split one pricing plan into three. Your scraper extracts what it can and drops the rest. Your competitive analysis is now based on an incomplete picture of their offering.

• Timezone and locale traps: This probably needs more attention than the others. The same competitor page can return different prices, currencies, or date formats depending on where the request originates. If your scraper runs from a US server but your competitor detects it and serves EU pricing, you’re tracking the wrong numbers.

What makes these failures dangerous is that you caused them yourself, unintentionally. Every defensive pattern you added to prevent crashes is exactly what turns a broken scraper into a quietly running one. No exceptions, no alerts, no red dashboards. Just weeks of nulls that nobody notices.

The rest of this article is about making that invisible failure visible again. Here’s a visual schema to get you introduced to the solutions proposed below:

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The Monitoring Layer: Your First Line of Defense

The first thing you should do to avoid such failures is to implement a monitoring layer. This is the part that the majority of scraping engineers skip. Because, you know how things go: the business needed the data yesterday, and you can’t lose time with monitoring…

But if you want your competitive scrapers to be resilient over time, you need to implement a monitoring system. A good one is made on top of four parts: output validation, structural fingerprinting, data freshness checks, and canary fields. Let’s discuss them!

How to Validate What Your Scraper Extracted

Managing HTTP status codes in your scrapers is not enough. You need to assert that the data you extracted is actually meaningful. The following example is taken from Stripe’s pricing page:

Here’s a practical validator using Pydantic that you can implement:

# In an activated virtual environment run: pip install httpx, bs4, pydantic

import httpx 
from bs4 import BeautifulSoup
from pydantic import BaseModel, field_validator, ValidationError
from typing import Optional
import re

# Define Pydantic model
class CompetitorPricingData(BaseModel):
    plan_name: str
    monthly_price: float
    currency: str
    features: list[str]

    @field_validator("monthly_price")
    @classmethod
    def price_must_be_positive(cls, v):
        if v <= 0:
            raise ValueError(f"Price must be positive, got {v}")
        return v

    @field_validator("currency")
    @classmethod
    def currency_must_be_valid(cls, v):
        valid_currencies = {"USD", "EUR", "GBP"}
        if v.upper() not in valid_currencies:
            raise ValueError(f"Unexpected currency: {v}")
        return v.upper()

    @field_validator("features")
    @classmethod
    def features_must_not_be_empty(cls, v):
        if len(v) == 0:
            raise ValueError("Features list is empty: possible extraction failure")
        return v

# Define scraper logic
def scrape_stripe_pricing() -> list[dict]:
    url = "<https://stripe.com/pricing>"
    headers = {
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/124.0.0.0 Safari/537.36"
        ),
        "Accept-Language": "en-US,en;q=0.9",
    }

    response = httpx.get(url, headers=headers, follow_redirects=True, timeout=15)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    plans = []

    # Get class name from selectors
    plan_cards = soup.select("div.HeroPricingSubcard")

    for card in plan_cards:
        container = card.select_one(".HeroPricingSubcard__container")
        if not container:
            plans.append({
                "plan_name": "",
                "monthly_price": 0.0,
                "currency": "USD",
                "features": [],
            })
            continue

        # Get all text nodes inside the container
        text_blocks = [t.get_text(strip=True) for t in container.find_all(True) if t.get_text(strip=True)]

        # Pull the first block that looks like a price
        price_text = ""
        for block in text_blocks:
            if re.search(r"[\\d]+[.,]?[\\d]*%?\\s*\\+?\\s*[€$£]?[\\d]*", block):
                price_text = block
                break

        price_match = re.search(r"[\\d]+\\.?[\\d]*", price_text.replace(",", "."))
        monthly_price = float(price_match.group()) if price_match else 0.0

        if "$" or "¢" in price_text:
            currency = "USD"
        elif "€" in price_text:
            currency = "EUR"
        elif "£" in price_text:
            currency = "GBP"
        else:
            currency = "UNKNOWN"

        # Use all text blocks as features
        features = text_blocks if text_blocks else []

        plans.append({
            "plan_name": "Standard",  # HeroPricingSubcard is the Standard card
            "monthly_price": monthly_price,
            "currency": currency,
            "features": features,
        })

    return plans

# Define validator logic
def send_alert(message: str):
    # Wire this to Slack, PagerDuty, email, etc.
    print(f"[ALERT] {message}")

def validate_scraped_data(raw_data: dict) -> Optional[CompetitorPricingData]:
    try:
        return CompetitorPricingData(**raw_data)
    except ValidationError as e:
        # Don't silently swallow this — alert immediately
        print(f"[VALIDATION FAILED] {e}")
        send_alert(f"Scraper validation failed: {e}")
        return None

if __name__ == "__main__":
    raw_plans = scrape_stripe_pricing()

    print(f"Found {len(raw_plans)} plan(s) on Stripe's pricing page.\\n")

    for raw in raw_plans:
        validated = validate_scraped_data(raw)
        if validated:
            print(f"[OK]   {validated.plan_name} — {validated.currency} {validated.monthly_price}/mo")
            print(f"       Features: {validated.features[:2]}{'...' if len(validated.features) > 2 else ''}")
        else:
            print(f"[FAIL] Raw data that broke validation: {raw}")

Here’s what this code does:

• Defines a Pydantic model: It leverages Pydantic to define the expected shape and rules of the scraped data. It enforces that the price is positive, the currency is one of USD, EUR, or GBP, and the features list is not empty. Any violation raises a ValidationError.

• scrape_stripe_pricing(): Fetches Stripe’s pricing page using the library httpx, parses the HTML with BeautifulSoup, and extracts pricing data using CSS selectors. If a container isn’t found, it appends a zeroed-out dict, which will intentionally fail validation downstream.

• validate_scraped_data(): Passes each raw scraped dict through the Pydantic model. On failure, it calls send_alert() instead of silently swallowing the error.

• send_alert(): A stub you can wire to your alerting stack (Slack, PagerDuty, email) in a production environment. Note that, for educational purposes, throughout the entire article, this is used to print alerts via the command line.

The result is the following:

Found 2 plan(s) on Stripe's pricing page.

[OK]   Standard — EUR 1.5/mo
       Features: ['1.5% + €0.25', '1.5% + €0.25']...
[OK]   Standard — EUR 2.5/mo
       Features: ['2.5% + €0.25', '2.5% + €0.25']...

If it’s not clear why this matters, here’s what your scraper “sees” when it’s pushed from another location:

The class is the same, but the data (the currency) is completely different! This is a common example of why you need validation that goes beyond a specific location.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Using DOM Fingerprints to Get Ahead of Structural Drift

This method is one of the most valuable things you can add to a competitive intelligence pipeline.

The idea is simple: periodically hash the DOM structure of the pages you’re scraping and store its value. When the hash changes, flag it for human review before your scraper breaks. Here is how you can implement it:

# In an activated virtual environment run: pip install requests

import hashlib
import httpx
from bs4 import BeautifulSoup
import requests

# Define parsing logic
def extract_structural_fingerprint(html: str, selector: str) -> str:
    """
    Extract a structural fingerprint from a specific section of the page.
    We hash the tag names and class names, NOT the content.
    This way, price changes don't trigger false alarms — only structural changes do.
    """
    soup = BeautifulSoup(html, "html.parser")
    container = soup.select_one(selector)

    if not container:
        return "CONTAINER_NOT_FOUND"

    # Build a structural signature
    structure = []
    for tag in container.find_all(True):
        classes = sorted(tag.get("class", []))
        structure.append(f"{tag.name}:{','.join(classes)}")

    fingerprint_str = "|".join(structure)
    return hashlib.md5(fingerprint_str.encode()).hexdigest()

# Compare hashes
def check_fingerprint(url: str, selector: str, stored_fingerprint: str) -> bool:
    """
    Returns True if the structure is unchanged, False if it has drifted.
    """
    response = requests.get(url, timeout=10)
    response.raise_for_status()

    current_fingerprint = extract_structural_fingerprint(response.text, selector)

    if current_fingerprint == "CONTAINER_NOT_FOUND":
        send_alert(f"[CRITICAL] Selector '{selector}' not found on {url}. Site may have been redesigned.")
        print("\\n[OK] Structure unchanged, safe to scrape.")
        return False

    if current_fingerprint != stored_fingerprint:
        send_alert(
            f"[WARNING] Structural change detected on {url} "
            f"(selector: {selector}). "
            f"Old: {stored_fingerprint[:8]}... New: {current_fingerprint[:8]}..."
        )
        print("\\n[WARNING] Structural change detected: hashes differ")
        print(f"  Old: {fingerprint[:8]}...")
        print(f"  New: {drifted_fingerprint[:8]}...")
        print("  → Scrape skipped. Human review required before next run.")
        return False

    return True

Below is a description of what this snippet does:

• extract_structural_fingerprint(): Parses the HTML, finds the target container via CSS selector, and builds a structural signature by iterating over every tag inside it. In this specific case, it records tag names and sorts class names, but deliberately ignores text content. That signature string is then hashed with MD5 and returned. If the selector finds nothing, it returns “CONTAINER_NOT_FOUND” instead of crashing.

• check_fingerprint(): Fetches the live page, computes the current fingerprint, and compares it against the stored one. Two failure cases are handled separately: the selector disappearing entirely (CONTAINER_NOT_FOUND) and the structure changing (hash mismatch). Both call send_alert() and return False: this signals to the caller to skip the scrape entirely.

• hashlib.md5(): Used to compress the structural signature into a short and comparable string. MD5 is not used here for security purposes; it’s used because it’s fast and collision-resistant enough for DOM comparison. Any change in tag names or class names produces a completely different hash.

In the case of a drifted fingerprint, the expected result is the following:

[STORED]  Fingerprint: 248363464b2da9b3814ea9a6dc5bd0df
[DRIFTED] Fingerprint: 7532ae7466c1f0fbc2c96471acd86ea7

[WARNING] Structural change detected: hashes differ
  Old: 24836346...
  New: 7532ae74...
  → Scrape skipped. Human review required before next run.

Note that this approach to DOM changes can be considered “classical” because it is fully based on finding a meaningful selector for the target page. The problem is that the DOM could change, but not the “meaningful” selector you choose. To improve it, you can target several selectors, or you can add a check on the whole body, but this would add more noise.

If you prefer trying a more “modern” approach that leverages LLMs to check on DOM changes, at The Web Scraping Club, we’ve already covered this in the article: “Beyond the DOM: A Practical Guide to Web Data Extraction with LLMs and GPT Vision”.

Using Canary Fields as a Heartbeat for Your Scraping Pipeline

A canary field is a field that you know changes frequently. This could be a “last updated” timestamp, a dynamic element, a session token in the page: you name it. You scrape it as a heartbeat, and if it stops changing, your pipeline is probably broken. Below is a snippet that checks on a canary field:

def send_alert(message: str):
    print(f"[ALERT] {message}")

def check_canary_field(
    current_value: str,
    stored_value: str,
    field_name: str,
    competitor: str,
    max_unchanged_hours: int = 48) -> None:
    if current_value == stored_value:
        send_alert(
            f"[WARNING] Canary field '{field_name}' for {competitor} "
            f"has not changed in over {max_unchanged_hours} hours. "
            f"Possible scraper failure or soft block."
        )
    else:
        print(f"[OK] Canary field '{field_name}' for {competitor} has changed. Pipeline looks healthy.")

When something changes, the expected output is the following:

[ALERT] [WARNING] Canary field 'last_updated' for <COMPETITOR NAME> has not changed in over 48 hours. 
Possible scraper failure or soft block.

Check the TWSC YouTube Channel

Building a Storage Architecture You Can Actually Trust

Many scraping engineers make the same mistake: they store only the processed output and throw away the raw HTML.

But in the case of competitive intelligence, that’s an issue that can cause you to work on weekends. Why? Because when a competitor changed their pricing 6 months ago, and you want to reconstruct the exact timeline, you need the raw data. Without it, history is gone forever.

The solution is a two-layer storage model: a raw and a processed layer. Here’s how the two relate:

             ┌─────────────────────────────────────────────────────┐
             │                     RAW LAYER                       │
             │  S3: raw/{competitor}/{page_type}/{date}.html.gz    │
             │  What's stored: full HTML, compressed, timestamped  │
             │  Why: so you can re-parse history if your logic     │
             │  changes or your scraper had a bug                  │
             └─────────────────────┬───────────────────────────────┘
                                   │ raw_s3_key (reference)
                                   ▼
             ┌─────────────────────────────────────────────────────┐
             │                  PROCESSED LAYER                    │
             │  SQLite/Postgres: pricing_history table             │
             │  What's stored: structured, queryable records       │
             │  Why: this is what your dashboards and alerts       │
             │  consume                                            │
             └─────────────────────────────────────────────────────┘

Basically, every record in the processed layer carries a raw_s3_key that points back to the exact HTML it was extracted from. That reference is what makes the whole architecture auditable.

The Raw Layer: Keep Every HTML Response

Store the full HTML or JSON response, timestamped, for every successful scrape. S3 is a cheap option to do so, and a year of raw HTML for a handful of competitor pages costs almost nothing. Below is an example you can implement in your pipeline:

# In an activated virtual environment run: pip install boto3

import boto3
import gzip
from datetime import datetime

def store_raw_response(
    html: str,
    competitor: str,
    page_type: str,
    bucket_name: str
) -> str:
    """
    Store the raw HTML response in S3 with a structured key.
    Compress it — HTML compresses extremely well (often 10:1).
    Returns the S3 key for reference in the processed layer.
    """
    s3 = boto3.client("s3")
    timestamp = datetime.utcnow().strftime("%Y/%m/%d/%H%M%S")
    s3_key = f"raw/{competitor}/{page_type}/{timestamp}.html.gz"

    compressed = gzip.compress(html.encode("utf-8"))

    s3.put_object(
        Bucket=bucket_name,
        Key=s3_key,
        Body=compressed,
        ContentEncoding="gzip",
        ContentType="text/html",
        Metadata={
            "competitor": competitor,
            "page_type": page_type,
            "scraped_at": datetime.utcnow().isoformat()
        }
    )

    return s3_key

def retrieve_raw_response(s3_key: str, bucket_name: str) -> str:
    """
    Retrieve and decompress a raw HTML response from S3.
    Useful for re-parsing historical data when your scraper logic changes.
    """
    s3 = boto3.client("s3")
    response = s3.get_object(Bucket=bucket_name, Key=s3_key)
    compressed_data = response["Body"].read()
    return gzip.decompress(compressed_data).decode("utf-8")

Notice the S3 structure: raw/{competitor}/{page_type}/{year}/{month}/{day}/{timestamp}.html.gz. This makes it trivial to list all scrapes for a specific competitor and page type within a date range, which is exactly what you need when reconstructing a pricing history.

Need help with your scraping project?

The Processed Layer: What Your Dashboards Actually Consume

The processed layer is your structured, queryable data: the thing your dashboards and alerts actually read. Every record here has two fields that make the whole architecture work:

• raw_s3_key, which links back to the exact HTML this record was extracted from.

• fingerprint, which captures the structural state of the page at scrape time.

The first one makes your data auditable. The second one tells you, later, whether the page had already changed when this record was written.

Below is a Python example to implement it:

import sqlite3
from dataclasses import dataclass
from datetime import datetime
from typing import Optional

@dataclass
class ProcessedPricingRecord:
    competitor: str
    page_type: str
    plan_name: str
    monthly_price: float
    currency: str
    scraped_at: datetime
    raw_s3_key: str  # The link back to the raw layer
    fingerprint: str  # The structural fingerprint at time of scrape

def store_processed_record(record: ProcessedPricingRecord, db_path: str) -> None:
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    cursor.execute("""
        CREATE TABLE IF NOT EXISTS pricing_history (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            competitor TEXT NOT NULL,
            page_type TEXT NOT NULL,
            plan_name TEXT NOT NULL,
            monthly_price REAL NOT NULL,
            currency TEXT NOT NULL,
            scraped_at TEXT NOT NULL,
            raw_s3_key TEXT NOT NULL,
            fingerprint TEXT NOT NULL
        )
    """)

    cursor.execute("""
        INSERT INTO pricing_history 
        (competitor, page_type, plan_name, monthly_price, currency, scraped_at, raw_s3_key, fingerprint)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?)
    """, (
        record.competitor,
        record.page_type,
        record.plan_name,
        record.monthly_price,
        record.currency,
        record.scraped_at.isoformat(),
        record.raw_s3_key,
        record.fingerprint
    ))

    conn.commit()
    conn.close()

A note on storage: this example uses SQLite, which is fine for a single scraper running on one machine. If you’re scraping multiple competitors in parallel or need concurrent writes, consider moving to Postgres (the schema remains the same).

A Concrete Example of Why the Raw Layer Saves You on Weekends

Imagine your competitor runs a Black Friday promotion in November and your scraper captures it. In January, your CEO asks: “When exactly did they drop prices, and by how much?”

With only the processed layer, you can answer that if your scraper was working correctly. But what if your scraper had a bug in November that caused it to extract prices incorrectly? With the raw layer, you can go back, fix the extraction logic, and re-parse the November HTML. Without it, that data is gone.

Here’s a utility function that makes re-parsing straightforward:

import boto3

def reparse_historical_data(
    competitor: str,
    page_type: str,
    start_date: str,
    end_date: str,
    bucket_name: str,
    new_parser_fn,
    db_path: str
) -> int:
    """
    Re-parse all raw HTML for a competitor/page_type within a date range.
    Useful when your extraction logic changes and you need to backfill.
    
    new_parser_fn: a callable that takes raw HTML and returns a list of dicts
    Returns the number of records reprocessed.
    """
    s3 = boto3.client("s3")
    prefix = f"raw/{competitor}/{page_type}/"

    paginator = s3.get_paginator("list_objects_v2")
    pages = paginator.paginate(Bucket=bucket_name, Prefix=prefix)

    reprocessed = 0

    for page in pages:
        for obj in page.get("Contents", []):
            key = obj["Key"]
            # Filter by date range based on key structure
            key_date = key.split("/")[3] + "-" + key.split("/")[4] + "-" + key.split("/")[5]
            if start_date <= key_date <= end_date:
                raw_html = retrieve_raw_response(key, bucket_name)
                parsed_records = new_parser_fn(raw_html)
                for record_data in parsed_records:
                    # Store with updated extraction, linked to same raw key
                    record = ProcessedPricingRecord(
                        **record_data,
                        raw_s3_key=key,
                        scraped_at=datetime.utcnow()
                    )
                    store_processed_record(record, db_path)
                    reprocessed += 1

    return reprocessed

Note that you’re not re-scraping anything. You’re just running your new parser against HTML you already have in storage. That’s the whole point of the raw layer.

Recovering From a Full Scraper Failure Without Losing History

Imagine this scenario: a competitor does a full site redesign, and your scraper is dead. This is not a hypothetical scenario: it happens more frequently than you can think, and most teams have no protocol for it.

Here’s a concrete recovery process.

How to Know Your Scraper Is Dead (Not Just Slow)

If you’ve implemented structural fingerprinting as discussed previously, you’ll know about the redesign before your scraper breaks. The fingerprint check will fail, you’ll get an alert, and you can investigate before storing bad data.

If you haven’t, and you’re reading this article after the fact, the first sign is usually a spike in validation failures or a sudden drop in scraped records. Consider implementing the following code:

import sqlite3

def detect_scraper_health(db_path: str, competitor: str, lookback_days: int = 7) -> dict:
    """
    Returns a health summary for a competitor's scraper over the last N days.
    Useful for spotting gradual degradation before it becomes a full failure.
    """
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    cursor.execute("""
        SELECT
            DATE(scraped_at) as scrape_date,
            COUNT(*) as total_attempts,
            SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successes,
            SUM(CASE WHEN status = 'validation_failed' THEN 1 ELSE 0 END) as validation_failures,
            SUM(CASE WHEN status = 'error' THEN 1 ELSE 0 END) as errors
        FROM scrape_log
        WHERE competitor = ?
          AND scraped_at >= DATE('now', ?)
        GROUP BY DATE(scraped_at)
        ORDER BY scrape_date DESC
    """, (competitor, f"-{lookback_days} days"))

    rows = cursor.fetchall()
    conn.close()

    return [
        {
            "date": row[0],
            "total": row[1],
            "successes": row[2],
            "validation_failures": row[3],
            "errors": row[4],
            "success_rate": round(row[2] / row[1] * 100, 1) if row[1] > 0 else 0
        }
        for row in rows
    ]

The expected result is something as follows:

Date         Total     OK   Val.Fail   Errors     Rate
--------------------------------------------------------
2026-01-15      10      0          0       10     0.0%
2026-01-14      10      2          5        3    20.0%
2026-01-13      10      5          4        1    50.0%
2026-01-12      10      8          2        0    80.0%
2026-01-11      10      9          1        0    90.0%
2026-01-10      10     10          0        0   100.0%
2026-01-09      10     10          0        0   100.0%

The degradation pattern is immediately readable. A success rate dropping from 95% to 60% over three days is probably a sign of a redesign in progress. A drop from 95% to 0% in a short time signals a full redesign, or a hard block.

Diagnosing What Actually Broke Before You Rewrite Anything

Not every structural change is a full redesign. Rewriting the major part of a scraper can cost you days if not weeks. To avoid that, you can write a script that checks what actually changed, pointing to meaningful parts of the target website:

from bs4 import BeautifulSoup

def triage_structural_change(
    url: str,
    selector: str,
    old_fingerprint: str,
    bucket_name: str,
    competitor: str,
    page_type: str) -> dict:
    """
    Diagnose the nature of a structural change.
    Returns a triage report to guide the recovery effort.
    """
    response = requests.get(url, timeout=10)
    current_html = response.text
    soup = BeautifulSoup(current_html, "html.parser")

    report = {
        "url": url,
        "http_status": response.status_code,
        "selector_found": soup.select_one(selector) is not None,
        "old_fingerprint": old_fingerprint,
        "new_fingerprint": extract_structural_fingerprint(current_html, selector),
        "page_title": soup.title.string if soup.title else "N/A",
        "recommendation": None
    }

    if not report["selector_found"]:
        report["recommendation"] = "FULL_REWRITE: primary selector is gone. Site likely redesigned."
    elif report["old_fingerprint"] != report["new_fingerprint"]:
        report["recommendation"] = "PARTIAL_UPDATE:selector exists but structure changed. Review child selectors."
    else:
        report["recommendation"] = "FALSE_ALARM: fingerprint mismatch may be transient. Re-check in 1 hour."

    # Store the current raw HTML for reference during rewrite
    store_raw_response(current_html, competitor, f"{page_type}_triage", bucket_name)

    return report
    
def print_report(report: dict) -> None:
    print(f"  URL:             {report['url']}")
    print(f"  Selector found:  {report['selector_found']}")
    print(f"  Old fingerprint: {report['old_fingerprint'][:8]}...")
    print(f"  New fingerprint: {report['new_fingerprint'][:8]}...")
    print(f"  Page title:      {report['page_title']}")
    print(f"  Recommendation:  {report['recommendation']}")

The expected output for the case of a full rewrite is the following:

	URL:             https://...
  Selector found:  False
  Old fingerprint: 4a7f92bc...
  New fingerprint: CONTAINER_NOT_FOUND
  Page title:      Pricing page
  Recommendation:  FULL_REWRITE: primary selector is gone. Site likely redesigned.

So, in this case, the script checks a specific URL of the target website. But the code can be, of course, generalized to more than just one. The same applies to the target selector.

Estimating How Much of the Gap You Can Recover From Raw Storage

Once you’ve rewritten the scraper, you have a gap in your data. If you’ve been storing raw HTML, you can partially fill it using a snippet like the following:

import boto3

def estimate_gap_coverage(
    competitor: str,
    page_type: str,
    failure_start_date: str,
    bucket_name: str) -> dict:
    """
    Estimate how much of the gap period can be recovered from raw storage.
    """
    s3 = boto3.client("s3")
    prefix = f"raw/{competitor}/{page_type}/"

    paginator = s3.get_paginator("list_objects_v2")
    pages = paginator.paginate(Bucket=bucket_name, Prefix=prefix)

    available_keys = []
    for page in pages:
        for obj in page.get("Contents", []):
            key = obj["Key"]
            key_date = "/".join(key.split("/")[3:6])
            if key_date >= failure_start_date.replace("-", "/"):
                available_keys.append(key)

    return {
        "gap_start": failure_start_date,
        "recoverable_snapshots": len(available_keys),
        "oldest_recoverable": available_keys[0] if available_keys else None,
        "newest_recoverable": available_keys[-1] if available_keys else None,
        "recommendation": (
            "Run reparse_historical_data() with your new parser to backfill."
            if available_keys else
            "No raw snapshots available for this period. Gap cannot be recovered."
        )
    }

# Print values
print(f"Gap start:              {result['gap_start']}")
print(f"Recoverable snapshots:  {result['recoverable_snapshots']}")
print(f"Oldest recoverable:     {result['oldest_recoverable']}")
print(f"Newest recoverable:     {result['newest_recoverable']}")
print(f"Recommendation:         {result['recommendation']}")

The expected result is:

Gap start:              2026-04-12
Recoverable snapshots:  3
Oldest recoverable:     <COMPETITOR_NAME>/pricing/2026/04/12/143022.html.gz
Newest recoverable:     <COMPETITOR_NAME>/pricing/2026/04/14/143022.html.gz
Recommendation:         Run reparse_historical_data() with your new parser to backfill.

As a final note on this recovery section, consider that, in practice, these three steps run in sequence:

• When something breaks, you start with detect_scraper_health() to confirm there’s actually a problem and understand how long it’s been degrading.

• Then you run triage_structural_change() to understand what broke. That tells you whether you need a full rewrite or a 10-minute fix.

• Once you’ve updated your scraper, you call estimate_gap_coverage() to see how much of the gap period you can recover from raw storage, and then reparse_historical_data() to actually backfill it. If estimate_gap_coverage() comes back with zero recoverable snapshots, that gap is gone, which is exactly why the raw layer exists in the first place.

Conclusion

Building competitive intelligence scrapers is not as easy as it seems for two reasons:

• Developing the scrapers is only half the effort.

• Keeping the scrapers honest 12 months from now probably needs even harder efforts than coding the scrapers themselves.

The layers presented in this article can make your competitive intelligence scrapers resilient through time (with the right adjustments for production environments). So, if you’re implementing this from scratch, don’t try to build everything at once. Add one piece at a time and take your time to verify everything works fine before you add the next step.

Then, you just need to analyze the data you scraped (maybe directly implementing a dashboard in Streamlit), and your colleagues in the business department can have sweet dreams: their competitive analytics are safe through time.

So, let us now: are your competitive intelligence scrapers time-resilient?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Kameleo Docker brings Kameleo’s anti-detect browser capabilities into a containerized setup, enabling production-ready automation with real fingerprinting and multi-profile isolation on Linux servers.

In this post, I’ll take a deep look at this solution and walk you through everything you need to know about it. By the end, you’ll understand what Kameleo Docker is, how its stealth browser approach works, how to set it up, and whether it’s actually worth trying.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

An Introduction to Kameleo Docker

Dig into the world of Kameleo Docker!

What Is Kameleo Docker?

Kameleo is an anti-detect browser engineered to make browser sessions look like real user devices. Instead of exposing a generic automation fingerprint, it creates realistic browser identities by spoofing hardware, browser, and environment signals such as WebGL, Canvas, fonts, screen resolution, and geolocation.

Kameleo Docker brings that same stealth stack into a self-hosted, containerized deployment model. Rather than relying on a desktop app, you can run Kameleo inside Docker on Linux or Windows servers, CI pipelines, VPSs, or Kubernetes environments.

Playwright, Puppeteer, and Selenium can connect to the container via CDP, meaning you can keep using your existing browser automation logic with minimal changes.

Further reading:

• Kameleo Docker product page

• Kameleo Docker docs

• Kameleo on Linux via Docker: What We Built, What Broke, What’s Next

Why Kameleo Docker Exists

Cloud servers, Kubernetes clusters, CI/CD pipelines, and VPS environments are overwhelmingly Linux-native, making Linux compatibility a practical requirement for automation and scraping teams.

For years, this created a problem for Kameleo users, as the solution only supported Windows and macOS. Because of that, teams wanting to run stealth browsers in production often had to rely on fragile workarounds.

Some deployed Windows virtual machines alongside Linux scraping stacks in AWS. Others used X11-over-SSH tunnels to remotely access browsers running on servers. These setups were difficult to maintain, resource-intensive, and far from ideal for scalable automation.

As explained in the product announcement blog post, Kameleo customers started to ask for a version of Kameleo that could run directly where their automation already lived.

As Barnabas Szenasi, founder and lead engineer at Kameleo, explained when I met him at Prague Crawl 2026:

“We could see from customer messages that a significant slice of our automation-first users were running Linux cloud servers and simply couldn’t use Kameleo at all... At Prague Crawl 2025, Tamas [Kameleo’s CEO] and I heard the same story from industry peers around the world: scraping pipelines were getting harder, and the need to run real browser environments instead of faking HTTP requests was growing fast.”

The Philosophy Behind the Project

From the beginning, Kameleo’s philosophy has been simple: masking quality matters more than shipping quickly.

After all, an anti-detect browser is only useful if it can convincingly behave like a real device. That’s why Kameleo relies on fingerprints sourced from real-world device traffic and continuously tested against modern anti-bot systems.

That same quality-first mindset also shaped the Docker project. According to founder and lead engineer Barnabas Szenasi, Linux support took longer than expected because the goal was never just to make Kameleo run in a container.

The objective was to reach the same masking quality users already expected on Windows and macOS. Shipping a functional but lower-fidelity Linux version would have compromised the product’s core standard.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Technical Architecture Behind Kameleo Docker

Now that you know why the project exists, let me explain how it works and the engineering behind it.

How Kameleo Docker Works

Kameleo Docker runs inside either:

• a Linux-based container (Ubuntu 22.04), or

• a Windows-based container (Windows Server Core 2022).

When you pull the image, Docker downloads the correct variant based on your container configuration (Linux containers are the default in most environments, including Docker Desktop). Regardless of the underlying platform, Kameleo exposes the same Local API for browser and profile management as the desktop app.

The process begins when you create a browser profile through the Local API. If you aren’t familiar with that concept, Kameleo profiles are reusable browser environments that bundle a complete browser fingerprint together with persistent browser state, such as cookies, browsing history, local storage, and bookmarks.

Profiles can also include user-defined settings like proxies, browser extensions, and startup preferences. Each profile is tied to a specific browser kernel and can be started, stopped, imported, or exported as needed.

Architecture Overview

Kameleo Docker separates browser execution from automation logic. The container hosts the stealth browsers, fingerprinting systems, and orchestration layer, while your automation scripts run independently on your machine, server, or orchestration platform.

At the center of the architecture is the Local API, exposed on port 5050. This API handles profile creation, fingerprint selection, browser startup, lifecycle management, and more.

Behind the API sit Kameleo’s browser kernels:

• Chroma: A Chromium-based engine.

• Junglefox: A Firefox-based engine.

These kernels are modified with engine-level masking patches and connected to Kameleo’s continuously updated fingerprint database (more on this later). Since they are exposed through the same API, you can switch between them without making any code changes.

When a profile starts, Kameleo launches a browser session with the configured fingerprint and settings. Playwright and Puppeteer can then connect to the running browser through a WebSocket endpoint via Chrome DevTools Protocol (CDP).

In other words, your automation script stays outside the Docker container. The browser behaves as if it were running locally, while execution, fingerprint masking, and browser management happen entirely inside the Docker container.

Persistent storage is handled through Docker volumes. Profile data, downloaded browser kernels, and runtime state are stored outside the container, allowing environments to be recreated without losing configuration or repeatedly downloading browser components. This makes deployments easier to scale, recover, and reproduce.

Trusted by teams running ad verification, web scraping, SERP tracking, and market research. Ethically sourced proxies, globally accessible, and fairly priced.

Get Started With DataImpulse

Core Features of Kameleo Docker

Time to explore the main features and capabilities provided by Kameleo Docker. For more information, read the official documentation.

Real Device Fingerprint Masking

Kameleo fingerprints are derived from real-world device traffic, not synthetic templates. Each profile represents a coherent combination of OS, browser, hardware signals, and behavioral characteristics.

All fingerprint surfaces are kept internally consistent, including Canvas, WebGL, audio context, screen resolution, and fonts.

Note: TLS fingerprint spoofing isn’t required, as Kameleo matches the browser kernel version precisely. The TLS stack remains the original, unmodified implementation shipped with the corresponding browser release.

The goal of the project isn’t to spoof everything, but to maintain realism across signals. That’s because overriding too many surfaces increases inconsistency risk, which detection systems can flag. For example, running a macOS fingerprint on a Windows host forces heavy compensation across system-level signals.

Proxy Integration and Geo Consistency

Each Kameleo profile can be assigned a dedicated proxy (including a rotating proxy), allowing IP-level isolation between browser identities.

Now, mismatches between IP geography and browser signals (language, timezone, WebRTC, and system locale) are a common detection vector. To address that, Kameleo provides automatic geo-location matching to align the browser’s settings with the geographic location of the selected proxy IP address.

Multi-Profile Isolation

Kameleo Docker is built around strict profile isolation, as each browser profile runs as a fully independent environment. This separation opens the door to safe multi-accounting (referred to as “account management” in Kameleo terminology). Thanks for this feature, you can operate multiple identities simultaneously without cross-contamination of session data or signals.

Linux-Specific Docker Image Features

Compared to the Windows-based container, the Linux version of Kameleo Docker includes several additional features. These include:

• Built-in VNC viewer: Allows you to monitor and interact with live browser sessions. This is especially useful for debugging automation, validating fingerprints, or troubleshooting rendering issues. You can access it through a browser on port 8080 or via native VNC clients such as RealVNC or TigerVNC on port 5900. For security reasons, it’s disabled by default.

• Browser-Based Kameleo GUI: A lightweight browser-based GUI on port 80 (reach it at http://localhost:80). Unlike the desktop app, it offers reduced functionality and is primarily intended for quick inspection, basic profile management, and monitoring.

• Optional GPU acceleration: The Linux container supports optional GPU acceleration for graphics-heavy workloads such as WebGL or canvas-intensive websites. Intel/AMD GPUs can be mounted through /dev/dri, while NVIDIA GPUs are supported through the NVIDIA Container Toolkit. When no GPU is available, Kameleo falls back to software rendering.

Getting Started With Kameleo Docker: Step-by-Step Guide

In this guided section, I’ll show you how to set up Kameleo Docker and use it for browser automation against the Scraping Course’s “JavaScript Rendering” page:

This is a sandbox environment for web scraping that simulates a real-world, JavaScript-rendered ecommerce page. It makes for a great testing target to validate the setup and see how Kameleo Docker behaves in a realistic automation scenario.

Requirements and Prerequisites

To get started with Kameleo Docker, make sure you have Docker installed and running locally. (Optionally, you need a Kameleo account if you want to run Kameleo Docker with authentication.)

For more details on supported operating systems and memory requirements, refer to the official documentation.

Since I’ll show how to use Kameleo Docker with Playwright in Python, to keep things moving, I’ll assume you already have a Python environment set up with Playwright and its dependencies installed.

To follow along with this tutorial section, I also recommend that you have:

• Basic Docker experience (running containers, mounting volumes, and using compose files)

• Familiarity with browser automation using Playwright.

Check the TWSC YouTube Channel

Step #1: Create a Kameleo Account

If you haven’t already, start by creating a Kameleo account. Fill out the sign-up form and enter the required information. Once registration is complete, a Free plan will already be activated:

Note that the Free plan is enough to use Kameleo Docker.

Important: Kameleo credentials are required for the container to authenticate successfully, download browser kernels, and start correctly.

Step #2: Start the Docker Container

With your account ready, the next step is to pull and run the Kameleo Docker image.

Remember that Kameleo ships as a multi-platform Docker image, supporting both Linux-based and Windows-based containers. To download and start the Linux container version of Kameleo, run:

docker run --platform linux/amd64 \
    --shm-size=2g \
    -p 5050:5050 \
    -e PAT='<YOUR_KAMELEO_PAT>' \
    -v kameleo-data:/data \
    kameleo/kameleo-app:latest

Note 1: If you run this command in PowerShell, replace “\” with the backtick “`” for multi-line commands.

Note 2: To launch the web GUI included in the Linux container of Kameleo Docker, add the -p 80:80 argument to your docker run command.

Here’s what matters in the above command:

• --platform linux/amd64 ensures Docker pulls the Linux-based image variant.

• --shm-size=2g is required for stable browser execution (the default Docker shared memory of 64MB is too small for browser execution).

• -v kameleo-data:/data creates a named volume that persists browser kernels and profiles across restarts.

• <YOUR_KAMELEO_PAT> is your Kameleo Personal Access Token (PAT), required to authenticate with your Kameleo account. Note that this argument is optional. You can skip the -e PAT=’<YOUR_KAMELEO_PAT>’ flag entirely if you want to try out Kameleo Docker without an account.

Below’s the output you should get:

The Kameleo Docker image should now be downloaded and launched on your system. Cool!

Step #3: Verify the Service

Once you run the image, Kameleo Docker will:

1. Start the Local API on port 5050.

2. Authenticate using your credentials.

3. Download required browser kernels (first run only).

To confirm everything is running correctly, visit the following URL in your browser:

http://localhost:5050/swagger

You should see the Swagger UI for the Kameleo Local API:

You’ll have a local Kameleo instance ready for automation with Playwright, Puppeteer, or Selenium. Great!

Step #4: Download the SDK and Create Your First Profile

Now that Kameleo Docker is running, you can interact with it through the APIs exposed at http://localhost:5050. The next step is to proceed with the usual Kameleo setup by creating a profile.

Assuming you already have a Python environment with Playwright installed,start by installing the Kameleo SDK:

pip install kameleo-local-api-client

Then, in your Playwright script, initialize the Kameleo API client and generate a browser profile:

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import CreateProfileRequest

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")
# Search for a real-world fingerprint and create a Kameleo profile based on it
fps = client.fingerprint.search_fingerprints(
    device_type="desktop",
    os_family="windows",
    browser_product="chrome",
    browser_version=">145",
)
profile = client.profile.create_profile(
    CreateProfileRequest(fingerprint_id=fps[0].id, name="twsc demo")
)

The above snippet connects the Kameleo API client from the SDK to the local Kameleo Docker APIs. It retrieves a realistic browser fingerprint from the database and creates a persistent browser profile called “twsc demo” based on it. In this case, the fingerprint profile is for a desktop Chrome browser (version >145) running on Windows.

Run the script above. If you started the Linux container of Kameleo Docker while mapping port 80 for the web GUI, then you’ll be able to see the “twsc demo” profile at http://localhost:80:

Step #5: Connect With Playwright

You can now connect Playwright to the Kameleo profile created above via CDP using the following WebSocket URL:

ws://localhost:5050/playwright/<KAMELEO_PROFILE_ID>

Achieve that by using Playwright’s connect_over_cdp() method on the target URL:

from playwright.sync_api import sync_playwright

# Kameleo profile creation...

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f"ws://localhost:5050/playwright/{profile.id}"
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Regular Playwright automation logic...

Wonderful! Playwright is attached to the browser session managed by Kameleo Docker. You can now automate it using standard Playwright APIs as if it were a regular local Chromium instance.

Step #6: Implement the Automation Logic

To achieve the scraping goal, begin by inspecting the page to study its DOM structure:

Then, apply the following Playwright logic (connected to a Kameleo profile) to automate scraping on the JavaScript-rendered page:

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f'ws://localhost:5050/playwright/{profile.id}'
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Open a new page
    context = browser.contexts[0]
    page = context.new_page()

    # Visit target site
    page.goto("https://www.scrapingcourse.com/javascript-rendering")

    # Where to store the scraped data
    products = []

    # Wait for products to render
    page.wait_for_selector(".product-item")

    # Locate all product items
    product_elements = page.locator(".product-item")

    for i in range(product_elements.count()):
        # Select the nth product
        product_element = product_elements.nth(i)

        # Extract the product data
        name = product_element.locator(".product-name").inner_text()
        price = product_element.locator(".product-price").inner_text()
        image = product_element.locator("img.product-image").get_attribute("src")
        link = product_element.locator("a.product-link").get_attribute("href")

        # Populate a product object with the scraped data
        product = {
            "name": name,
            "price": price,
            "image": image,
            "url": link
        }
        # Append it to the products list
        products.append(product)

The above snippet instructs the controlled browser to visit the target page, waits for product elements to load, then iterates through each product DOM node to extract structured fields (name, price, image, URL) and stores them in a Python list for downstream processing.

The Kameleo-powered automation script is almost complete. Only one step remains!

Step #7: Stop the Kameleo Profile

Normally, in a Playwright scenario, you would need to call browser.close() to terminate the browser session and release its resources.

In Kameleo, that’s not required. Instead, you only need to call:

client.profile.stop_profile(profile_id=profile.id)

The above line of code sends a close command to the browser via CDP. Once the browser actually stops, the Kameleo profile is terminated, too. This ensures that all resources associated with both the browser and the running profile are properly released.

Step #8: Run the Script

The final Playwright automation script, connecting via CDP to the stealth browser instance exposed by Kameleo Docker, will contain:

# pip install playwright kameleo-local-api-client

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import CreateProfileRequest
from playwright.sync_api import sync_playwright

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")
# Search for a real-world fingerprint and create a Kameleo profile based on it
fps = client.fingerprint.search_fingerprints(
    device_type="desktop",
    os_family="windows",
    browser_product="chrome",
    browser_version=">145",
)
profile = client.profile.create_profile(
    CreateProfileRequest(fingerprint_id=fps[0].id, name="twsc demo")
)

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f'ws://localhost:5050/playwright/{profile.id}'
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Open a new page
    context = browser.contexts[0]
    page = context.new_page()

    # Visit target site
    page.goto("https://www.scrapingcourse.com/javascript-rendering")

    # Where to store the scraped data
    products = []

    # Wait for products to render
    page.wait_for_selector(".product-item")

    # Locate all product items
    product_elements = page.locator(".product-item")

    for i in range(product_elements.count()):
        # Select the nth product
        product_element = product_elements.nth(i)

        # Extract the product data
        name = product_element.locator(".product-name").inner_text()
        price = product_element.locator(".product-price").inner_text()
        image = product_element.locator("img.product-image").get_attribute("src")
        link = product_element.locator("a.product-link").get_attribute("href")

        # Populate a product object with the scraped data
        product = {
            "name": name,
            "price": price,
            "image": image,
            "url": link
        }
        # Append it to the products list
        products.append(product)

    # Print the scraped products
    for product in products:
        print(product)

# Stop the Kameleo profile
client.profile.stop_profile(profile_id=profile.id)

Execute the script, and you should see output similar to this:

Notice how the script successfully scraped product data from the JavaScript-rendered page.

Once execution completes, open the Kameleo web GUI, and you’ll notice that the “twsc demo” profile is now marked as “TERMINATED”:

That doesn’t mean the profile is gone forever. Quite the opposite!

Kameleo profiles are reusable, meaning you can retrieve and start them again later to continue the browsing session with the same fingerprint and persisted state. I’ll cover exactly how to do that in a dedicated FAQ.

Mission complete! You just learned how to use Kameleo Docker for Playwright automation. With very similar logic, you can automate Puppeteer, Selenium, or any other CDP-compatible solution, in both Python and JavaScript.

Pricing Model

Kameleo Docker is included across all plans at no additional cost. You get the same core limits (concurrent browsers, number of profiles, and browser usage time) regardless of whether you run the desktop app or the containerized version. So, take a look at the official pricing page for more information.

Anti-Bot Performance Benchmarks

To test Kameleo Docker, I ran a simple script against one page protected by each major anti-bot detection system. The results are shown below:

Note: All tests were performed locally using my ISP’s residential IP address.

As shown above, in this basic experiment, Kameleo Docker achieved a 100% success rate. In contrast, Playwright consistently failed in headless mode and, in some cases, also struggled in headful mode.

Need help with your scraping project?

Final Thoughts on Kameleo Docker

What stood out when I met Barnabas at Prague Crawl 2026 (see you next year 😉) was the clear passion the team has for the project, along with their focus on quality and continuous improvement.

At the same time, when testing new products, especially technical and complex ones like Kameleo Docker, you usually stumble across bugs or unexpected behavior. I can confidently say that this wasn’t the case at all here. Everything ran smoothly from the beginning, and I didn’t encounter any issues…

On top of that, the benchmark results are promising, and I didn’t notice any significant performance lag. Thus, my honest takeaway is simple: if you’re looking for a production-ready, containerized stealth browser, or you’re simply passionate about automation and scraping, consider giving Kameleo Docker a try!

In this article, I covered what the project is about, what it offers, how it works, and how to use it. As always, remember to use Kameleo Docker only for legal and ethical web scraping and automation. Until next time!

FAQ

Is Kameleo Docker different from the desktop app?

Kameleo Docker differs from the desktop app mainly in deployment. Instead of a local GUI, it runs as a containerized service that runs on both Linux and Windows servers. Feature parity is largely preserved, including profiles, fingerprinting, and browser engines.

Can I reuse already created profiles in Kameleo Docker?

Yes! Profiles can be reused by retrieving the full list of profiles, filtering by name (or ID, if you know it), and then starting the desired profile. For example, to reuse the “twsc demo” profile, write:

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import ProfileLifetimeState

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")

# Fetch all available profiles
profiles = client.profile.list_profiles()

# Find the profile with the specific name
target_name = "twsc demo"
profile = next((p for p in profiles if p.name == target_name), None)

# Check if the profile was found
if profile:
    # Start the existing profile if it isn't already running
    if profile.status.lifetime_state != ProfileLifetimeState.RUNNING:
      client.profile.start_profile(profile.id)

Can I use Kameleo Docker in my CI/CD?

Kameleo Docker fits naturally into CI/CD pipelines by running as a disposable, reproducible container in build or test stages. You can spin up browsers on demand, run automated flows, and tear them down after execution. Configuration is typically handled via Docker Compose.

Does Kameleo Docker support proxy integration?

Yes! HTTP, HTTPS, and SOCKS proxies can be configured at profile creation time, as explained in the documentation.

Can Kameleo Docker scale to thousands of browsers?

Kameleo Docker supports horizontal scaling through standard orchestration tools. You can run multiple containers across clusters using Kubernetes or AWS ECS, each managing independent browser instances.

How Does Firefox Automation Work in Kameleo Docker?

Kameleo Docker supports Firefox-based automation through the Junglefox engine. Because Playwright cannot connect directly to Firefox-based sessions, Kameleo provides a pw-bridge helper that acts as a compatibility layer. This component translates Playwright connections into the correct browser session, allowing standard automation scripts to run unchanged while still using Firefox-based fingerprint profiles.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share