Archive - The Web Scraping Club

THE LAB #71: Sending Scrapy logs to RabbitMQ

Saving the logs of your distributed scraping architecture to your database

Dec 19 •

Pierluigi Vinciguerra

How We Scraped Global Hotel Data to Track Economic Trends

Tracking Tourism and Economic Flows with Global Hotel Data Scraping

Dec 17 •

Andrea Squatrito

How to Parse JSON with Python: A Beginner-Friendly Guide

Tips and tricks for handling JSON in your scraping operations

Dec 15 •

Pierluigi Vinciguerra

THE LAB #70: Advanced logging in Scrapy

How to extract the most meaningful metrics from your scrapers

Dec 12 •

Pierluigi Vinciguerra

How Scraping the Web Became an Expensive Business

How the industry moved from gathering low-hanging fruits to an expensive engineering challenge.

Dec 10 •

Andrea Squatrito

Optimizing Proxy Usage for Large-Scale Scraping

How to cut costs of scraping with some simple rules

Dec 8 •

Pierluigi Vinciguerra

THE LAB #69: Building a dashboard for your scrapers with Grafana

Visualizing the operations of your Scrapy spider with interactive dashboards

Dec 5 •

Pierluigi Vinciguerra

Scraping The Inflation

Measuring inflation might seem the domain of economists and statisticians, but it is increasingly reliant on vast networks of web scraping—a practice as…

Dec 3 •

Andrea Squatrito

Making Playwright scrapers undetected with open source solutions

Let's see the latest libraries for making your Playwright scrapers unstoppable

Dec 1 •

Pierluigi Vinciguerra

November 2024

#nojs-banner { position: fixed; bottom: 0; left: 0; padding: 16px 16px 16px 32px; width: 100%; box-sizing: border-box; background: red; color: white; font-family: -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 13px; line-height: 13px; } #nojs-banner a { color: inherit; text-decoration: underline; } This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts