In my experience, any site that needs to show changing data, like a product catalog or prices that update often, almost never puts that data directly in the HTML.
Instead, the site loads a basic shell and then fetches the live content from an internal API, usually as JSON, which the frontend then renders. E-commerce sites rely on this approach the most, since stock and prices change all the time and nobody wants to redeploy a page for every update. In fact, this is how modern frontends work.
Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.
This actually works in our favor. Internal APIs are much more stable targets than the HTML around them. If the site gets a redesign, all your selectors might break, but the JSON endpoint usually stays put because the frontend still needs it. They’re also easy to spot, since they use regular HTTP in the browser, so you can watch them and see exactly what comes back. The real challenge is finding the right request and figuring out how to reach it past any anti-bot protection.
I always start by hand, in the browser, because that’s the best way to build a mental model of the target and keep your skills sharp. After that, I move to code, since clicking around DevTools for every new site gets tedious, and a small interception script can turn a twenty-minute manual search into a quick automated run. For this example, I’ll use Harrods (https://www.harrods.com/en-it/) as the target. By the end, we’ll be pulling the full product catalog straight from the API the frontend uses.
Doing it by hand first
Open DevTools, go to the Network tab, and filter to Fetch/XHR. That filter drops the images, fonts, scripts, and CSS, and leaves you with the requests the page makes to move data around. On a well behaved site you reload the page, watch a handful of JSON calls show up, and one of them is obviously the product list.
From identifying internal API calls to analyzing structured responses, LLMs are transforming web research. anyIP powers the access layer with residential and mobile IPs built for modern web-data workflows.
Harrods doesn’t make it that easy, and the reason is worth understanding because it’s common. When you load a category page like women-new-in, the products are already in the HTML. The page is server side rendered, so the first response comes back with the grid baked in and no separate product API call fires. If you only look at the initial load, you’ll conclude there’s no API and reach for HTML parsing.
The API is there. It just doesn’t fire until the page needs new products it doesn’t already have. So you paginate. Click to page two, and now the frontend has to go get sixty products it doesn’t have in memory, so it makes the call. This is the habit that pays off most when you hunt internal APIs. You interact with the page and watch what fires. Paginate, filter, sort, scroll. The data call shows up when the page needs data.
Do that on Harrods with the Network tab open and one request stands out from the analytics noise, a POST to /api/search/1/indexes/*/queries. That path is a giveaway. It’s the Algolia query API, and we’ve pulled product data out of Algolia before (https://substack.thewebscraping.club/p/scraping-algolia-endpoints). Harrods proxies it through its own domain instead of calling Algolia directly, which turns out to matter, but the shape is familiar.
That’s how I do it manually. It works, and honestly, it teaches you more about the target than any tool. But it’s slow, and if you need to check ten sites in an afternoon, it just doesn’t scale. So I automate the search instead.
What Harrods actually is
Before I run any tool, I like to map out the layers, since each one is a possible failure point.
The storefront is a Nuxt application, which we can see from the /_nuxt/builds/meta/*.json calls in the traffic. Nuxt server renders the first page and then hydrates into a single page app, which is exactly why the product grid is in the initial HTML but pagination is a client side fetch. Commerce runs on SCAYLE, visible from the hrd-prd.middleware.api.scayle.cloud and hrd-live.cdn.scayle.cloud hosts. Product search is Algolia, proxied through harrods.com/api/search. Currency and pricing for the Italian storefront are handled by Global-e, which shows up as the gepi.global-e.com calls.
On top of all that, there’s Akamai Bot Manager. This is the layer you have to get past just to load the page, and it decides if you can even use an automated browser. I’m not going to reverse engineer Akamai’s sensor here. That’s another story. Instead, I’ll sidestep it using a stealth browser and a warm-up navigation, then focus on the data. For this walkthrough, the real target is the JSON API layer. Akamai is just the wall I need to climb to get there.
Building the interceptor
The core of the tool is a handler attached to the browser’s response event. For every response, I check the content type, keep the JSON ones, drop the analytics noise, and save the body. Here’s the handler, from intercept.py:
def make_handler(captured, bodies_dir):
seen = set()
async def on_response(response):
try:
headers = await response.all_headers()
except Exception:
return
ctype = headers.get("content-type", "").lower()
if not any(m in ctype for m in JSON_MARKERS):
return
if is_noise(response.url):
return
request = response.request
base = response.url.split("?")[0]
key = (request.method, base)
if key in seen:
return
seen.add(key)
try:
raw = await response.body()
except Exception:
raw = b""
...
print(f" [json] {request.method} {response.status} {base} ({len(raw)} B)")
return on_response
For the content type filter, I use application/json, application/ld+json, text/json, and anything ending with +json. To cut out the noise, I block hosts that are almost always analytics or consent, Google, Facebook, OneTrust, Segment, and similar, so they don’t drown out the real endpoints. I deduplicate by method and path, since hitting a paginated endpoint five times is still just one endpoint. Every response I keep gets its body saved to disk, so I can check the payloads later.
Attaching it is one line, right after I open the page:
There are two details that make this work on a real site.
First, you need a warm up. If you go straight to a deep URL like women-new-in, Akamai will block you, since you’re asking for an inner page without the cookies a real browser would have collected. So I always load the homepage first, let Akamai set its _abck and bm_sz cookies, wait a bit, and only then go to the category page in the same session.
if warmup:
print(f" warmup -> {warmup}")
await safe_goto(page, warmup, settle_ms * 2)
if await safe_goto(page, url):
await drive(page, scrolls, settle_ms)
Second, there’s pagination. From the manual check, I know the product API call only happens when the page needs products it doesn’t already have. Just navigating to ?page=2 isn’t enough, since that triggers another server-rendered load. What you need is a client-side click, just like a real user would do. The most reliable way is to trigger the click through the DOM. Playwright’s normal click kept timing out on the pagination control.
async def click_through(page, selectors, clicks, settle_ms):
for selector in selectors:
for n in range(clicks):
count = await page.locator(selector).count()
if not count:
print(f" click skipped: {selector} not in DOM")
break
await page.evaluate("() => window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(1000)
try:
await page.eval_on_selector(selector, "el => el.click()")
print(f" js-click [{n + 1}] {selector} (matches={count})")
await page.wait_for_timeout(settle_ms)
except Exception as exc:
print(f" click failed: {selector} -> {type(exc).__name__}")
break
Running it
Everything here runs from the shared venv, where Playwright (1.55.0), Camoufox (0.4.11), and curl_cffi (0.14.0) were already installed.
First the control. Plain Playwright, headless, straight at the category page:
playwright._impl._errors.Error: Page.goto: net::ERR_HTTP2_PROTOCOL_ERROR
at https://www.harrods.com/en-it/women-new-in
That’s Akamai dropping the connection at the HTTP/2 layer, before any HTML is served. It didn’t like the TLS and connection fingerprint of an unpatched automated browser. Worth noticing that this is a different failure than a block page. There’s no “Access Denied” body to read, the socket just closes. The control confirms plain Playwright is a non starter on this target, which is the whole reason the control exists.
Now Camoufox, with the homepage warm up and a click through to page two:
This time the page loads, the title comes back as Designer Women New In | Harrods IT, and the harness prints the JSON calls as they land. Stripping the Akamai sensor beacon and the third party hosts, here’s what Harrods itself serves as JSON:
GET 200 /_i18n/10acb0d1/en-gb/messages.json (54186 B)
POST 200 /api/rpc/getUser
POST 200 /api/rpc/getBasket -> basket
POST 200 /api/rpc/getWishlist -> key,items
POST 200 /api/rpc/getHashedString -> hash
POST 200 /api/rpc/getExternalIdpRedirect -> harrods
GET 200 /_nuxt/builds/meta/d3043d4e-...-631f954ba023.json
POST 200 /api/search/1/indexes/*/queries -> results (488909 B)
The /api/rpc/* calls are the storefront’s own internal RPC endpoints for things like the basket, wishlist, and user session. These are interesting and worth mapping if you’re modeling the site, but they’re not what I was after. The last line is the one I wanted: a 488 KB POST to the Algolia proxy, keyed on results, and it only showed up after clicking to page two. On the first load, it wasn’t there, just like I saw in the manual check.
Reading the payoff
Here’s what came back in that response:
top keys: ['results']
result[0] keys: ['hits', 'nbHits', 'offset', 'length', 'facets', 'query', 'index', 'queryID', ...]
index: prod_harrods.com | nbHits: 1199
hits on this page: 60
Sixty products in one call, out of 1199 in the category, from the index prod_harrods.com. Each hit carries what you’d actually want:
You get product id, reference key, name, brand, gender, URL, images, category hierarchy, and price as a map of currencies in minor units. For example, GBP: 79800 means £798.00. This is exactly the same data you see in the product grid, but now you can store it directly without having to parse the HTML.
The reusable part is the request itself. Here’s the POST body the frontend sent:
If you decode the params, it’s all clear. The facetFilters with categoryPageIdentifier:Women New In limits the query to this category. offset=60&length=60 handles pagination, so page two starts at offset sixty. Change the offset and you can walk through the whole category, sixty products at a time.
One thing is clearly missing here. There’s no x-algolia-application-id or x-algolia-api-key, which a direct Algolia call always carries. That’s because we’re not calling Algolia, we’re calling harrods.com/api/search, and the proxy injects the credentials server side. We don’t need to find or rotate an Algolia key, we just POST to the Harrods path with the cookies our browser already earned.
Replaying it to page the whole catalog
Continue reading this post for free, courtesy of Pierluigi Vinciguerra.
