The Web Scraping Club

The Web Scraping Club

THE LAB #110: Why Shopee is so hard to scrape

Every route into Shopee hits a different wall. I mapped all of them and got past none.

Pierluigi Vinciguerra's avatar
Pierluigi Vinciguerra
Jul 16, 2026
∙ Paid

Usually, when I dig into an app’s request signing, I like to end with something practical you can use, like a script, an offline signer, or at least a way to forge a header. This time, though, things turned out differently, and I think that’s actually more interesting. I tried to tackle Shopee from every angle I know: the native library inside the app, the JSON API behind the website, you name it. Every route hit a different wall. So here’s my map of those walls, why they hold up, and, to be honest, why I couldn’t get product data out of any of them.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan


I’m picking up where I left off in a previous investigation, so I won’t repeat all the details here. Back then, I pulled libshopeeaegis.so (one of Shopee’s native crypto libraries) into Ghidra and rebuilt it in Python, byte for byte. That was a straightforward win: just an integer switch over standard AES, SHA, and HMAC, once you get past the intentionally confusing symbol names. But as I mentioned at the end of that article, Aegis is just a general-purpose toolbox, not the actual library that signs API requests. The real signer is a different beast, and that’s what I’m looking at now.

What actually gates the API

Capture any Shopee API request, strip the anti-fraud headers, replay it, and the backend answers with an HTTP error and code 90309999. That code is the whole story of this article. It is the anti-fraud rejection, and by the end you will have seen it come back from the app, from a plain HTTP client, and from two different real browsers.

Two headers carry the protection. On the Java side, two interceptors in okhttp, the HTTP client library the Android app is built on, attach them: com.shopee.app.network.antifraud.b and .d, which name themselves SecurityNewSapInterceptor and SecurityNewSapPostInterceptor. The first header is af-ac-enc-sz-token, a constant for the session:

af-ac-enc-sz-token: ghHLT8yNv0Xd8t9C2/t6Sw==|uJEC5q...JfDCw4ImZS5+gWyOO5Dr...8bd|rsGw3AWMR7Ryh7Jo|08|1

The right foundation to keep a smooth scraping workflow. anyIP helps teams collect web data more consistently with high-quality residential and mobile IPs worldwide.

Explore anyIP Proxies →


This token is a pipe-delimited string: an encrypted device fingerprint, a 16-byte wrapped key, an 81-byte ciphertext, a 12-byte nonce, and two version markers. It’s computed once per session and then reused. The second header, x-sap-ri, comes with a set of signatures keyed by short hex strings, and these change every request since they’re computed over the request itself. Both headers are generated by libshpssdk.so, a hefty 4.7 MB native library. The tricky part is the per-request signature, since that’s the bit you can’t just precompute and reuse.

Finding the signer, and deciding not to port it

libshpssdk.so exports exactly one symbol, JNI_OnLoad. Everything else is hidden, which tells you the native methods are registered at runtime with RegisterNatives, the Java Native Interface (JNI) call an app uses to bind native functions to Java methods, rather than exported by name. So we catch the registration as it happens. On arm64, RegisterNatives is a slot in the JNIEnv function table at byte offset 0x6b8. We hook that slot spawn-gated, before the app’s own code runs, and read out every method the library registers. The agent is in /shopee_sap_oracle/agent_oracle.js.

We are after one JNI signature. A request signer takes the request and returns headers, which in JNI terms is two byte arrays in and a string out, ([B[B)Ljava/lang/String;. One method matches. It is vuwuuwvw, registered at libshpssdk+0xc7ebc, on the obfuscated class com.shopee.shpssdk.wvvvuwwu, a name we recovered from the jclass argument of the registration call through java.lang.Class.getName().

When I opened that address in Ghidra, I found the function itself barely does anything. It just marshals its arguments and passes them to another function, along with a 440-byte blob and a table of function pointers. That’s the classic shape of a bytecode interpreter: the blob is the program, the pointer table is the opcode handlers, and the real logic lives in the bytecode, not in any native function I can easily read. This is where my earlier approach hits a wall. Last time, I could just read the primitive bodies and match them to AES and SHA. Here, there are no such primitives. To reimplement the signer, I’d have to reverse and interpret the whole virtual machine (VM), a project that would take weeks and is on the same level as breaking DroidGuard, Google’s own anti-abuse VM. So instead of porting it, I decided to just call it directly.

The one thing that works: the oracle

The signer is a Java method on a class we already resolved, so we drive the app’s own code. The agent exposes one RPC (remote procedure call) export that takes a URL and a body, turns them into the two byte arrays the method expects, and returns whatever the signer returns. The driver (/shopee_sap_oracle/frida_oracle.py) spawns the app, waits for the signer to register, and signs requests we choose. Re-running it against the current app, the signer registers in three seconds and answers:

[SIGN] url=https://mall.shopee.ph/api/v4/native/homepage body={"test":1}
   3c7b6059 = LR4te3nzTKb0RcbDqGs1bsXb0pB=
   71356940 = uMaRB6hpxBCq1BwZ2bFqGXjGFow=
   a0e0dcb3 = WFiutLJdM2C0t5Lr4yaoPr9CLn4=
   eaab58ec = 3BJOkkGhS53kj19RA2QqVLfC1m8sZFLSYm8W8S+Kl3yHU+vuzvb+...   (~2 KB)
   x-sap-ri = d9e7576a26efc4c9896e6716017c0e63cf0cae2c4c560ed7dd03

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here


That’s the win, and I want to be clear about what it actually means. I can now produce the exact headers the app would attach for any request I choose, without needing to understand how the signer works under the hood. But before I trust these headers, it’s worth looking closely at the output, because it gives you a sense of how tough it’ll be to reuse them. The structural checks I’m about to show run on the captured values using /shopee_sap_oracle/analyze_headers.py. No device needed.

The short values look like base64 of a 20-byte digest, which is the length of HMAC-SHA1, so the easy reading is that they are keyed hashes and that is where a first pass leaves them. The easy reading is wrong. Decode seven of them as standard base64, and five come back with non-zero bits in the final padded position, which a conforming base64 encoder never produces from raw bytes:

  6m6r94xGLKAjTMYNMepdMiLQpSB=  len=20  NON-canonical (low2=1)
  wLJO02yz2I+FAZETVwC0aTCflyB=  len=20  NON-canonical (low2=1)
  1iwMbNiaPma2XLT7ToRs4jgiasE=  len=20  canonical
  47WhCSbrTx1r5FdFFlgtpHPN0OO=  len=20  NON-canonical (low2=2)
  LzCPR2/LYpvqP/z1HBXtuzj+8dQ=  len=20  canonical
  vmPAmagMmOlffOGIWF8MwJEBuNT=  len=20  NON-canonical (low2=3)

Check the TWSC YouTube Channel


So these aren’t just plain base64-encoded digests. Either the alphabet is shuffled or there’s some transformation after encoding, and that’s exactly the kind of detail that trips up anyone trying to reproduce the signature with HMAC-SHA1 and a standard encoder. The keys themselves are interesting too: within a session, no two requests share a key, and those eight-hex identifiers aren’t the actual header names you see on the wire. Plus, endpoints with a body return four entries, while GETs without a body return three. There’s a whole mapping layer from the signer’s per-request keys to the real header names, and I never managed to reverse that part.

Then there is x-sap-ri itself. It is 26 bytes: a leading counter, a three-byte session ID, an eight-byte middle field, a 01 marker at byte twelve, and a thirteen-byte tail. The counter is the interesting part, because it is coarser than it looks. In the fresh capture, both requests came back with counter d9 and session e7576a in front, and only the middle and tail differed:

d9 e7576a 26efc4c9896e6716 01 7c0e63cf0cae2c4c560ed7dd03
d9 e7576a b4c3b40323380413 01 d56d3f5c7eaef45395425b9720

So the leading byte isn’t just a simple counter that increments with each request. What actually separates one request from another is hidden in the middle and tail fields, and the token is tightly bound to the specific request through those parts. That binding is the first reason why a replay fails, and it’s where the real obstacles begin.

App wall one: the replay is out of sequence

The obvious next step was to sign a real product request using the oracle and try replaying it from outside the app. I picked a phone case from Shopee’s public site (shop_id=20396998, item_id=5420033304), had the oracle sign its product detail URL, attached the session’s af-ac-enc-sz-token and a cookie I grabbed with a one-shot header hook, and replayed the request with curl-cffi. The driver for this is /shopee_sap_oracle/replay.py. Here’s what Shopee sent back:

[*] oracle signed 4 headers: ['5cfcca02', 'c382fc01', 'cf1e6149', 'x-sap-ri']
==== REPLAY RESULT ====
HTTP 403
{"5":false,"2":false,"0":2,"3":90309999,"error":90309999, ...}

The status code changes depending on the endpoint, but the error code is always the same. Having a valid signature is necessary, but it’s not enough. This replay was always going to fail for two reasons: first, the x-sap-ri binding ties the token to the device’s own request sequence, so any injected, out-of-sequence request gets rejected, even with a fresh signature. Second, I put the signer’s per-request-keyed values under those same eight-hex keys, but I never actually confirmed that’s how the app names them on the wire. For a clean external replay, you’d need both of those right, and I could only be sure about one.

App wall two: the RASP cancels everything

The way to remove both doubts is to stop replaying and make the request from inside the app, where the sequence, the device state, the TLS, and the real header names are already coherent. The app builds its API calls on okhttp, so we went looking for its OkHttpClient at runtime with Java.choose. The probe is in /shopee_sap_oracle/crash_trace.js. There are more than a dozen live instances, and you spot the API client by its interceptors. The antifraud layer is not among the application interceptors where we first looked. It sits lower, among the network interceptors, which run after okhttp has set up the real connection, so it signs the request that is actually about to go on the wire.

As always, the code that will be used for the python reimplementation we’re showing now can be found in our GitHub repository reserved for paying users, inside the folder 110.SHOPEE-PART2.

User's avatar

Continue reading this post for free, courtesy of Pierluigi Vinciguerra.

Or purchase a paid subscription.
© 2026 The Web Scraping Club SRL · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture