Let’s consider a common scraping scenario. You’ve been working on a new scraping project for days on a target website that is a heavily protected e-commerce. You launch the scraper to test it, but your first run gets blocked within a few minutes. This is where you try using a datacenter proxy pool because, you know, they are cheap. But the scraper is blocked again.
Then you get access to a residential proxy pool, and the scraper works.
If you’ve been there, you probably moved on without asking why. The adage is simple: if it works, don’t touch it!
But that “why” matters more than you think because residential proxies aren’t a kind of black magic. They’re a specific technical architecture on top of which your scrapers work. And as with any architectural choice, you have to make decisions on purpose and based on specific parameters.
In this article, you’ll get a complete technical picture of residential proxies, starting from how IP addresses are classified, how the proxy infrastructure routes your traffic, what residential proxies are, and their common types. In the end, you’ll also discover the ethics behind residential IP sourcing and how to choose ethical providers with a practical decision framework.
Let’s get into it!
Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.
An IP address (Internet Protocol address) is a numerical label assigned to every device connected to a network that uses the Internet Protocol for communication. On the practical side, the IP protocol serves two core functions:
Host or network interface identification.
Location addressing.
The current dominant versions are IPv4 and IPv6:
IPv4 addresses are 32-bit integers, expressed in dotted-decimal notation as four octets ranging from 0 to 255 (for example, 203.0.113.47), yielding a theoretical address space of ~4.3 billion unique addresses.
Due to exhaustion of this space, IPv6 was introduced, using 128-bit addresses expressed in colon-separated hexadecimal groups (for instance, 2001:0db8:85a3:0000:0000:8a2e:0370:7334), offering a space of ~3.4 × 10³⁸ addresses.
IP addresses aren’t handed out randomly by providers: there’s a strict hierarchy that they must respect. Specifically, a global authority called IANA divides the total pool of IP addresses into large blocks and distributes them to regional bodies known as RIRs (Regional Internet Registries). These RIRs then assign smaller blocks to ISPs (your home internet provider) and cloud providers (like AWS or Google Cloud), who finally assign individual IPs to end users and servers.
Why does this matter for scraping? Because this entire chain is publicly documented, and anyone can look up who owns a given IP address using tools like WHOIS (a directory of IP ownership records). This means a target website can instantly tell whether your request is coming from a residential IP, a datacenter IP, or a known proxy or VPN provider.
This is precisely why residential proxies are so valuable in web scraping: their IPs are indistinguishable from those of regular home users, making them much harder for anti-bot systems to flag.
A proxy server is an intermediary host that sits between your client and the target web server, forwarding requests and responses on your behalf:
Proxy servers’ architecture
Without a proxy, the TCP connection to the target server is established directly from your machine. With a proxy, your client opens a connection to the proxy, which opens a separate connection to the target, and the target sees the proxy’s IP address as the request origin. This way, your real IP never appears in the target’s access logs or network layer.
This IP substitution is the fundamental value proposition of a proxy in a scraping context. Every request your scraper sends carries a source IP and, as covered above, target servers actively use that IP to make trust decisions. By routing through a proxy, you externalize your network identity. This way, the anti-bots on target servers evaluate the reputation, geolocation, and classification of the proxy’s IP, not yours.
At the network level, a proxy is simply a process listening on a TCP port, capable of accepting inbound connections and initiating outbound ones. It maintains two separate TCP connections for each proxied request: one with your client and one with the target. These connections are independent: they can use different protocols, different keep-alive settings, and the proxy can pool or reuse the outbound connection across multiple client requests.
When routing traffic through a proxy, the protocol determines how the proxy handles your traffic at the application and transport layers. This has real implications for scraping purposes. The main protocols are the following:
HTTP/HTTPS: Proxies that use these protocols operate at the application layer (OSI Layer 7). When your scraping client sends a request through an HTTP proxy, it constructs a full HTTP request and sends it to the proxy server, which forwards it to the target on your behalf. For plain HTTP traffic, the proxy reads the request in full. For HTTPS traffic over an HTTP proxy, the client issues a CONNECT method request, instructing the proxy to establish a raw TCP tunnel to the destination. Once the tunnel is open, the TLS handshake occurs directly between your client and the target server. This means that the proxy becomes a blind pipe and cannot decrypt the payload. This is critical to understand: an HTTP proxy handling HTTPS via CONNECT does not perform SSL termination (unless it’s explicitly a man-in-the-middle proxy with a custom CA cert injected).
SOCKS5: Is a lower-level protocol operating at the transport layer (OSI Layer 5). Unlike HTTP proxies, SOCKS5 is protocol-agnostic. This means that it simply relays raw TCP (and optionally UDP) streams without any awareness of the application-layer protocol being tunneled. After a brief handshake, the proxy establishes a TCP connection to the destination and shuttles bytes in both directions without inspection. This makes SOCKS5 suitable for any TCP-based protocol, not just web traffic.
One operationally significant detail to consider is DNS resolution. With SOCKS5, the domain name is forwarded to the proxy for resolution, meaning DNS queries originate from the proxy’s network, not yours. This prevents DNS leaks, which can expose your real network location even when your traffic routes through a proxy. HTTP proxies also resolve DNS on the proxy side for CONNECT-tunneled requests, but behavior can vary by implementation.
Now, here’s a question for you: for scraping purposes, which protocol is the best? Well, for most scraping use cases, either protocol is functionally equivalent from the target server’s perspective, since both can tunnel TLS. SOCKS5 is generally preferred for its flexibility and slightly lower overhead, but HTTP proxies are often easier to drop into existing HTTP client configurations without additional dependencies.
Tired of stitching together proxies, browsers, and parsers just to scrape one page? Gaffa handles the whole stack for you, with the control to fine-tune every step when you need it.
A residential proxy is a proxy server whose outbound IP address is assigned by a consumer Internet Service Provider (ISP) to a real, physical household device like a home router, laptop, smartphone, smart TV, or similar. The word “residential” refers directly to the ISP classification of the IP: these are addresses allocated to residential customers, and they appear as such in public IP intelligence databases.
How Residential Proxies Work
The operational model of a residential proxy network involves three distinct parties: the proxy provider (the company offering proxy servers on the market), the exit node (the residential device), and the scraping client (you).
The exit node is a real consumer device whose owner has installed software that, in exchange for some compensation, allows the provider to route third-party traffic through their internet connection. This software runs as a background process and registers the device with the provider’s orchestration infrastructure. The device’s residential IP is thus made available in the provider’s pool.
The provider’s infrastructure is the backbone that ties everything together. It consists of:
Agateway layer: A set of proxy endpoints that your scraping client connects to. These endpoints are hosted on standard data center infrastructure and serve as the entry point into the residential network.
Anorchestration/routing layer: The internal system responsible for selecting which residential exit node to use for a given outbound request, based on criteria like target geolocation, session persistence requirements, and node availability.
The exit node pool: The fleet of residential devices registered with the provider, each maintaining a persistent connection back to the orchestration layer.
Thanks to this infrastructure, the scraping requests flow like that:
Your scraping client connects to the provider’s gateway endpoint and authenticates.
The gateway receives your request and passes it to the orchestration layer, which selects an available residential exit node from the pool based on your session’s parameters.
The orchestration layer forwards the request to the selected exit node over the persistent internal tunnel.
The exit node’s software initiates the actual outbound TCP connection to the target server from the residential device’s IP.
The target server receives a request from what appears to be a residential IP, processes it, and returns the response.
The response travels back through the same path in reverse: exit node → orchestration layer → gateway → your client.
This is why, from the target server’s perspective, the only visible connection is the one from the residential exit node. Your client’s IP, the provider’s gateway IP, and the internal routing infrastructure are entirely invisible.
Residential proxies are segmented into distinct types based on two primary axes: IP persistence (how long you hold the same IP across requests) and targeting granularity (how precisely you can select the exit node’s location).
This section reports and describes the common residential proxy categories that are based on those two axes.
Rotating Residential Proxies
This is the most common type. With rotating proxies, the provider’s orchestration layer assigns a new exit node—and, therefore, a new IP—for each outbound request, or at a configurable interval. From your client’s perspective, you connect to the same gateway endpoint every time, but each request exits from a different residential IP. This is the default behavior on most residential proxy platforms and is the correct choice for large-scale scraping where you need to distribute requests across a wide IP footprint to avoid per-IP rate limits. Belos is a schema that shows, at a high level, how this process works:
Rotating residential proxies schema
The rotation can be triggered in two ways, depending on the provider:
Per-request rotation: A new IP is selected for every single HTTP request.
Timed rotation: The same IP is held for a configurable duration before rotating.
In the context of web scraping, the primary tradeoff is that rotating proxies make stateful interactions difficult. If a target website tracks session state via server-side session IDs or cookies tied to an IP, switching IPs mid-session will break the session. So, for purely stateless scraping cases like fetching product pages, search results, and public data, rotation is ideal.
Sticky Residential Proxies
Sticky proxies allow you to hold the same residential IP for an extended, configurable period. This typically ranges from a few minutes up to 30 minutes, depending on the provider, and is bounded by the exit node’s availability online. The session is maintained by passing a session identifier in your proxy credentials, which instructs the orchestration layer to pin your requests to the same exit node for the duration of the session.
This type is essential for scraping workflows that require statefulness, like multi-step form submissions, login-based scraping, checkout flows, or any target that correlates sequential requests by IP and invalidates sessions on IP change. The risk is that if the exit node goes offline mid-session (the residential device disconnects from the network, which happens regularly on consumer connections), the session is broken, and the provider will typically reassign you to a new IP. This requires your scraper to handle session recovery; otherwise, your pipelines will break.
Static Residential Proxies
Static residential proxies (often known as ISP proxies) occupy a middle ground. These IPs are registered to an ISP (giving them a residential-looking ASN and WHOIS classification) but are hosted on stable data center infrastructure, rather than actual consumer devices. The provider procures these IPs directly from ISPs and hosts them on servers they control.
The key advantage over standard rotating residential proxies is stability and speed. Since the IP is hosted on a server rather than a consumer device, it doesn’t go offline unpredictably, latency is lower, and bandwidth is not constrained by a household connection. You get a fixed, persistent residential-classified IP that behaves like a data center proxy in terms of reliability.
The tradeoffs are:
Cost: These are among the most expensive proxy types.
Increasing detectability: As anti-bot vendors are building databases of known static residential/ISP proxy IP ranges, this kind of proxy works as a cat-and-mouse race to escape anti-bots.
Mobile Proxies
Mobile proxies are a specialized subtype where the exit node is a mobile device connected via a cellular network, and the IP belongs to a mobile carrier’s ASN. These are distinct from standard residential proxies because mobile IPs carry a specific characteristic: mobile carriers use large-scale carrier-grade NAT (CGNAT), meaning a single public IP is shared among potentially thousands of concurrent mobile users.
This has a profound implication for anti-bot systems: blocking a mobile carrier IP risks blocking thousands of legitimate users simultaneously, making mobile IPs extremely difficult for targets to block without significant collateral damage. As a result, mobile proxies are the hardest proxy type to block and command the highest price per GB in the market. They are most valuable for targets with aggressive anti-bot systems, typically heavily protected e-commerce sites, social platforms, and ad verification use cases.
The tradeoff is latency and bandwidth variability. The exit node is on a cellular connection, which is subject to signal quality, network congestion, and higher round-trip times compared to a wired residential or data center connection.
Residential Proxies Types: A Summary Comparison
The table below summarizes the subtypes of residential proxies, each with its own specific characteristics:
How Residential Proxies Are Sourced and Why It Matters
The technical sophistication of residential proxy networks is surely interesting for techies. But what’s also interesting is where the exit nodes actually come from. This is because understanding the sourcing model is not just an ethical exercise. It has direct operational implications for scraping professionals because the legality and reputational standing of the proxies you use can affect your liability exposure and the long-term reliability of your infrastructure. It also has implications for companies working in highly regulated markets, where specific certifications matter more than anything else. We’ve seen recently how bad things can go from one day to another if your proxy provider is not fully compliant on both IP sourcing side and customer onboarding.
As introduced previously, providers build their residential IP pools by recruiting real consumer devices to act as exit nodes. The mechanisms for doing this fall into a spectrum that is described in the subparagraphs below.
SDK Monetization
The most widespread IP sourcing method is SDK embedding. In this model, a residential proxy provider operates a B2B program where mobile app developers and software vendors integrate the provider’s SDK into their applications. In exchange, the developer receives a revenue share or a flat fee per active device. When an end user installs the app, the SDK runs in the background and enrolls the device’s internet connection into the proxy pool.
The provider’s SDK typically does the following:
Establishes a persistent outbound connection to the provider’s orchestration infrastructure.
Monitors device state to only proxy traffic when the device is charging and connected to Wi-Fi (to avoid draining battery or mobile data, which would generate user complaints).
Throttles bandwidth consumption to stay below the threshold of user noticeability.
Handles NAT traversal to make the device reachable for outbound proxy sessions.
The apps embedding these SDKs span a wide range. Just to name a few:
Free VPN clients.
File utility apps.
Weather apps.
Games.
So, the SDK is not the app’s core function: it’s the monetization layer that the developer has grafted on.
Opt-In Peer Networks
Some providers operate a peer model, where users knowingly and explicitly enroll their devices in exchange for a direct benefit. This typically is free access to a premium VPN or bandwidth credits. The consent flow clearly explains that the user’s connection will be used to route third-party traffic.
This is the most ethically defensible sourcing model, as the value exchange is explicit and the user retains full awareness of what their device is being used for.
Ethical Considerations
The ethical landscape of residential proxy sourcing is genuinely complex, and it deserves some treatment to help you discern the right proxy provider to choose.
The Consent Problem
The foundational ethical issue is informed consent. When a residential device is used as an exit node, its owner’s internet connection carries traffic for a third party. This means that their IP address becomes associated with requests they did not initiate, to targets they may never have interacted with. If a scraping client uses that exit node to send requests that the target considers abusive, the device owner’s IP is the one that appears in server logs. The potential consequences for an unknowing participant range from having their IP blacklisted by services they use personally to, in extreme cases, being flagged by automated fraud or abuse detection systems.
This is why true informed consent in this context is very important. Also, for deep ethical reasons, informed consent would require a clear, plain-language explanation that the user’s internet connection will relay unknown third-party traffic, that their IP will be the visible origin of that traffic, and that this may affect their IP’s reputation.
Bandwidth and Resource Consumption
Enrolling a device in a proxy pool consumes the device owner’s bandwidth. In many markets, this resource is metered and has a real financial cost. In mobile-heavy markets where Wi-Fi and cellular data are tightly managed household resources, this is a non-trivial extraction of value from the device owner. This is another reason why proxy providers offer rewards to IP owners.
The Downstream Use Problem
Even when sourcing is fully and clearly consensual, the ethical chain does not end at the exit node: it extends to what the proxy traffic is used for. This is because a residential proxy provider cannot fully control what its clients scrape. This means that the same infrastructure used by a legitimate price monitoring service can be used for credential stuffing attacks, ad fraud, scalping bots, or scraping protected personal data.
For this reason, responsible providers implement terms of service enforcement, traffic monitoring, and client vetting. As a scraping professional, you are a link in this chain and carry some responsibility for the downstream ethics of the traffic you generate.
Legal Exposure
The legal status of residential proxy usage varies by jurisdiction and use case. In the United States, the Computer Fraud and Abuse Act (CFAA) has been interpreted broadly to potentially cover unauthorized access to computer systems. And whether bypassing a website’s IP-based blocks via residential proxies constitutes “unauthorized access” has been the subject of significant litigation.
In the European Union, the GDPR introduces additional complexity when scraped data includes personal information. So, note that using residential proxies does not insulate you from these legal frameworks. It only changes the network-layer attribution of the requests.
What to Look For in a Residential Proxy Provider
If you’re new to the scraping industry and need to get residential proxies, you may be overwhelmed by the offers that are on the proxy market. There are a lot of providers out there, and choosing the best one for you may be hard.
If you want to evaluate providers not only based on budget, the suggestion for you is to primarily make ethical considerations. If you don’t know where to start, indicators of a responsible residential IP sourcing model include the following:
Published transparency reports detailing how exit nodes are recruited.
Clear opt-in documentation and publicly accessible SDK disclosure pages.
Active terms of service enforcement against prohibited use cases.
Direct compensation models for exit node participants.
Compliance certifications. Some of the most ethical providers have pursued SOC 2 or ISO 27001 certifications, which indicate a structured approach to operational governance.
The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Conclusion
Residential proxies are easy-to-use tools, but may be hard to understand. The gap between “it works” and “I know why it works” is where most engineers live, and it’s fine until it isn’t.
Understanding the full stack changes how you make decisions for your scraping projects. Why? Because it changes how you pick a provider, how you evaluate proxy types for a given target, and how you think about the ethical and legal weight of the infrastructure you’re routing traffic through.
So, now that you know how this infrastructure actually works, let us know: does it change how you’d pick proxies for your next scraping project?
Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.
