Session Stickiness at Scale - Insights from DataImpulse
Building a Proxy Rotation Layer That Mimics Real Users
This is a guest post written by the DataImpulse team. For an independent benchmark of proxy prices, visit our Proxy Pricing Benchmark tool.
You buy proxies from a provider with the biggest IP pool you can find. You rotate IPs for every request. You randomize User-Agent strings. You connect to proxy servers from various regions. You add delays between requests. You expect to smoothly scrape the necessary data in no time, as you have made everything so your scraper does not leave any traces behind and is impossible to get caught. The next thing you see usually happens around request 150-300 - this is blocks. CAPTCHAs, error 403, redirects to honeypot pages serving you wrong data. The whole pipeline crumples down.
If this situation is painfully familiar, you have probably let session stickiness out of sight while building a proxy rotation layer - a common mistake. This article explains in detail how to rotate proxies to mimic real users so your scraper remains a reliable data source even at scale.
Why careless IP rotation may speed up bot detection
When you realize a website has blocked you out, the first instinct is to add more IPs - maybe buy proxies that perform better and are more adapted for dealing with detection systems. You try to add more geo-diversity. In other words, you use the same approach as before, just scale it. Sometimes it helps a bit; however, sometimes, blocks are back even faster.
The mistake here is that you are looking for a problem in the wrong place. It is not the lack of IP diversity, but the lack of session consistency.
Modern anti-bot systems like Cloudflare, Akamai Bot Manager, DataDome, or PerimeterX do not rely on the IP address as a primary method of bot detection. Instead, they consider a lot of signals and model session behavior. In such circumstances, your rotating efforts, if not strategic, can only make things worse and produce behavioral signals that are bot-like, not user-like. More IPs are not a fix for that problem. Modeling behavioral signals of genuine users is.
The anatomy of bot detection
In order to build a truly stealthy scraper with a rotation layer that gets the job done, it is important to understand first how detection happens - what exactly can give you away.
So, there are four layers of different signals that anti-bot systems analyze simultaneously and make a decision whether to let you be or deny access.
Layer 1 - Network
IP reputation, ASN classification, type of IP (residential, mobile, or datacenter), and provider fingerprints belong to that layer. Also, if a proxy or VPN IP is known, it would also be shown here.
This layer is the one developers usually optimize the most for. Ironically, this is the least important layer for security systems. That is why you fail when following scraping tutorials that offer to simply set as big an IP pool as possible and rotate addresses for every request. It is not because you have done something wrong or your proxies are bad, but because this approach does not work for scraping at scale.
Layer 2 - TLS/HTTP fingerprints
HTTP clients’ fingerprints are nearly as unique as an IP address. TLS handshake leaves quite a trace with its cipher suite order, list of supported extensions, and elliptic curves. HTTP/2 creates another signal with header prioritization, window size, and frame ordering. The combination of TLS and HTTP/2 fingerprints is often enough for security systems to figure out whether it’s human-generated or bot-generated traffic, even without considering signals from other layers.
Widely used scraping libraries, for example, curl, requests, httpx, aiohttp produce distinct, detectable fingerprints. Python is a popular choice for building scrapers now; still, if you use its requests library, a calibrated WAF would spot it from TLS handshake alone, User-Agent headers be forgotten, whatever they say.
Layer 3 - Behavioral
Requests cadence, referrer chain integrity, cookie handling, navigation patterns, and timing distributions are evaluated here. Even the delays between requests, if they are too precise and equal, can fuel suspicions, as humans do not make requests with a machine’s precision. Real users’ referrers have logic to them. Human users’ cookies are consistent and show that they came from a search engine page. They do not disappear mid-session or mid-navigation.
Layer 4 - Session Graph
This layer evaluates whether the sequence of cookies, tokens, page flow, and state transitions within the session makes sense for a human user. That is a layer most scrapes fail to pass, as not so many developers pay attention to coherence. Cause even if stand-alone signals look normal, it is important that they make sense when evaluated together. For example, a user visits an e-commerce website. Their session token would show that. Their viewport size would stay the same throughout the visit. They will go between pages with different time intervals, reflecting reading and decision time.
Coherence is the key that opens or closes a door for your scraper. And one proxy rotation at the wrong time can reveal your scraper’s true identity.
Session stickiness: a true solution
In the context of a proxy architecture, session stickiness is a condition when all requests that belong to the same logical session are routed via the same IP address. At the very least, it must be the same subnet, geo location, and ASN cluster.
Now, let’s define what a session may mean, cause in terms of scraping, there are several options possible.
Authentication session
The most obvious case. You log in, receive a session token, and proceed to the data. If your IP changes somewhere between the login request and the first data request after authentication, to the target server, it looks like a user logged in from Paris and then suddenly that same user is asking for data from Seoul. Session invalidation or re-verification is guaranteed.
Stateful pagination
This type is subtler and catches even sophisticated scrapers off guard. The thing is that many modern applications track your pagination state on a server’s side - in other words, a server “remembers” a page you are currently at and ties it to your IP. If the IP address changes between pages 3 and 4, the server loses this track. It may either block you or return incorrect data.
Multi-step forms
Cases when you need to simulate checkouts, survey scrapers, or lead generation cases are the most session-sensitive. The state that was established at the beginning is carried forward across the rest of the steps. A mid-process IP change is a red flag in such a scenario; detection systems will react immediately.
To sum up, it is not random IPs you need, but sessions and a rotation strategy that considers session coherence.
Building the rotation layer
Time to get to work. Remember: rotating session objects, not IP addresses.
The session object model
A session object, in simple words, is a set of signals that make a set of requests look like they originated from a real human user.
@dataclass
class ProxySession:
session_id: str
proxy_endpoint: str # host:port or provider session token
cookies: dict = field(default_factory=dict)
headers_profile: dict = field(default_factory=dict)
geo: str = "" # e.g. "pl,us,en"
asn: str = "" # e.g. "12345"
created_at: float = field(default_factory=time.time)
last_used: float = field(default_factory=time.time)
request_count: int = 0
flagged: bool = False
def is_expired(self, ttl_seconds: int = 600, max_requests: int = 200) -> bool:
age = time.time() - self.created_at
return self.flagged or age > ttl_seconds or self.request_count >= max_requests
The headers_profile is not just a User-Agent string. It is a set of HTTP headers that are used for each and every request of one session: User-Agent, Accept-Language, Accept-Encoding, Accept, sec-ch-ua, sec-ch-ua-platform, sec-ch-ua-mobile. They must stay the same for every request. If one request is coming as if from Chrome browser version 108, and another one from version 124, detection systems will notice a mismatch. Even without knowing exactly that you are a scraper, such a mismatch is enough to block you.
Sticky session assignment
Any scraper must use the same session object within one session - this is a rule. So the session assignment must not be random, but based on that rule to prevent any mix-ups.
import hashlib
import threading
from typing import Dict
class SessionPool:
def __init__(self, proxy_provider):
self.provider = proxy_provider
self._sessions: Dict[str, ProxySession] = {}
self._lock = threading.Lock()
def _make_key(self, job_id: str, target_domain: str) -> str:
raw = f"{job_id}:{target_domain}"
return hashlib.md5(raw.encode()).hexdigest()[:12]
def get_or_create(self, job_id: str, target_domain: str) -> ProxySession:
key = self._make_key(job_id, target_domain)
with self._lock:
session = self._sessions.get(key)
if session is None or session.is_expired():
session = self.provider.new_session(geo=self._infer_geo(target_domain))
self._sessions[key] = session
session.last_used = time.time()
session.request_count += 1
return session
def _infer_geo(self, domain: str) -> str:
# Simplified — in production, map TLDs/domains to geo requirements
tld_map = {".pl": "PL", ".de": "DE", ".co.uk": "GB", ".fr": "FR"}
for tld, geo in tld_map.items():
if domain.endswith(tld):
return geo
return "US" # Default
Notice that the session key is determined by job_id + target_domain, so the scraper sticks to the same session until it expires. The session expiration is also based on 2 rules - TTL (sessions older than 10 minutes automatically get refreshed) and request count (after 200 requests, the session gets refreshed to prevent statistical fingerprinting from long-lived sessions).
Geo-tied rotation
Another life hack is to rotate IPs within the same geo or ASN cluster even when a session has expired, and you are spinning to the new one. Even between sessions, sudden changes (like jumping from Tokyo residential IP to Toronto residential IP) are noticeable and suspicious. When that happens, it looks more like a VPN switch, not a user returning.
Result-oriented proxy providers offer sticky sessions. In such a case, a session token has a location backed into it, and the provider guarantees that the traffic will originate as if from that location during a session’s lifetime.
Header profile coherence
Not only must IP remain the same throughout the session. If you rely on the same address, but the header profile changes from request to request, it is like a “Hello, I’m a scraper” sign for detection systems.
HEADER_PROFILES = [
{
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"sec-ch-ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
},
# Add more profiles reflecting real browser market share distributions
]
One session - one profile. No changes mid-session.
Retry logic
Retry logic is another thing that can blow your cover. When a request fails due to some reason and the target server instantly receives another one from the same origin, it most certainly means bots are in the game, as humans do not act with such light speed. An even less obvious trap is retrying with a different session object when your current session simply needs a brief backoff.
The better approach is to retry while relying on the same session object and a slight delay. If you receive a clear block signal, like 407, 429 errors with specific headers, or CAPTCHA, switch to the new session.
def fetch_with_retry(
url: str,
pool: "SessionPool",
job_id: str,
target_domain: str,
max_retries: int = 3,
) -> requests.Response:
failures: list[str] = []
for attempt in range(max_retries):
session = pool.get_or_create(job_id, target_domain)
try:
response = make_request(url, session)
if response.status_code in (407, 429) or is_captcha(response):
session.flagged = True
failures.append(
f"attempt {attempt}: session {session.session_id} "
f"blocked (status={response.status_code})"
)
if attempt < max_retries - 1:
base = min(2 ** attempt * 2, 30)
time.sleep(base * (0.8 + 0.4 * random.random()))
continue
return response
except (requests.Timeout, requests.ConnectionError) as e:
failures.append(f"attempt {attempt}: {type(e).__name__}: {e}")
if attempt < max_retries - 1:
base = min(2 ** attempt * 2, 30)
time.sleep(base * (0.8 + 0.4 * random.random()))
raise MaxRetriesExceeded("All attempts failed:\n " + "\n ".join(failures))
There is an important difference: TransientError (which includes network hiccup and timeout) retries with the current session. BlockedError marks the session as flagged, which in turn triggers is_expired() and forces a new session creation.
Scaling concurrent sessions
At a small scale, the session pool model works just fine. However, if your scraper runs on 50+ machines simultaneously, you need all of them not to create new sessions each, but to stick to the one.
A Redis-backed session pool is a solution. Basically, Redis is a shared notepad that lives on a server and that every machine can read and write to, so that there is no chaos in sessions. Store each ProxySession object (serialized as JSON) with a TTL key:
import redis
import json
r = redis.Redis(host='localhost', port=6379, db=0)
def store_session(session: ProxySession, ttl: int = 600):
key = f"session:{session.session_id}"
r.setex(key, ttl, json.dumps(session.__dict__))
def load_session(session_id: str) -> Optional[ProxySession]:
data = r.get(f"session:{session_id}")
if data:
return ProxySession(**json.loads(data))
return None
If a proxy provider offers sticky sessions, your Redis only needs to track the session token, not the exit node state. A provider handling IP consistency and you dealing with cookies and headers is the preferred architecture for effective scaling because building and maintaining a distributed session pool is good 2-4 weeks of work. You need developers to do it - and add ongoing maintenance as websites update their detection systems.
Testing rotation layer
To be sure everything works fine, verify your rotation layer. Send 10 requests to see whether all of them really go via the same IP address.
import requests
def verify_session_stickiness(session_token: str, n: int = 10):
proxy = f"http://user__sessid.{session_token}:pass@gw.dataimpulse.com:823"
seen_ips = set()
for i in range(n):
r = requests.get("https://httpbin.org/ip", proxies={"https": proxy})
ip = r.json()["origin"]
seen_ips.add(ip)
print(f"Request {i+1}: {ip}")
print(f"\nUnique IPs: {len(seen_ips)}")
assert len(seen_ips) == 1, f"FAIL: Session leaked across {len(seen_ips)} IPs"
print("PASS: Session sticky across all requests")
verify_session_stickiness("your_session_token_here")
To verify headers, use https://httpbin.org/headers and compare headers across requests. Any differences in User-Agent, Accept-Language, or sec-ch-ua between requests of the same session are a problem.
If you want a deeper test, use bot.sannysoft.com or pixelscan.net. Those sites show what detection systems see - WebDriver flags, inconsistent browser APIs, JA3 anomalies, timezone/language mismatches. You see what exactly is wrong and at which layer. This is an easier, faster, and cheaper way than constant tries and errors.
Final thoughts
Scrapers that remain undetected are not the ones with the largest IP pools. They are the ones that mimic real users’ behavior the most persuasively - when the IP, the TLS fingerprint, the header profile, the cookie state, and the navigation flow all together create traffic patterns that are typical for a genuine human visitor.
IP rotation is necessary, but it is a tactic, while session architecture is a strategy. When you adopt that approach, you feel the difference - in lower block rates and higher data accuracy, in how much less firefighting your team goes through every time security systems update. Scalability is another benefit.
Also, there is an undeniable tendency for detection systems to develop, and so must your scrapers do. Rotation that is based on sessions, not single IPs, allows for that development.
If you want to skip the “build infrastructure from scratch” part, DataImpulse is an ethical proxy provider that offers both rotating and sticky sessions. You can set the duration of a session manually - up to 120 minutes is possible. DataImpulse (dataimpulse.com) provides 90M+ of residential, mobile, and datacenter enterprise-grade addresses in 195 locations with advanced targeting options (extra costs) to help maintain session coherence. Web scraping, SERP tracking, price monitoring, and other sensitive scenarios are among its use cases. Prices start from $1/GB - for non-expiring traffic and 24/7 human support.