The Web Scraping Club

The Web Scraping Club

Home
Consulting
Proxy pricing benchmark
The Lab
Club Deals
Archive
About

The Lab

THE LAB #80: Scraping food delivery data
Use both the website and the mobile apps to get data from food and grocery delivery data
 • 
Pierluigi Vinciguerra
THE LAB #79: Use Cursor as web scraping assistant with MCP servers
Add MCP Servers to Cursor for increasing our web scraping capabilities
 • 
Pierluigi Vinciguerra
THE LAB #78: Building a Web Scraping Knowledge Assistant with RAG - Part2
Optimizing the content storage and creating a CLI for our assistant
 • 
Pierluigi Vinciguerra
THE LAB #77: Building a Web Scraping Knowledge Assistant with RAG
How to include scraped data in your AI assistant
 • 
Pierluigi Vinciguerra
THE LAB #76: Bypassing Kasada With Open Source Tools In 2025
How to bypass Kasada protected websites without paying a cent
 • 
Pierluigi Vinciguerra
THE LAB #75: Building self healing scrapers with AI
How can we use LLMs to analyze HTML and fix our web scrapers?
 • 
Pierluigi Vinciguerra
THE LAB #74: Running scrapers on GitHub Actions
Save money and time by using GitHub infrastructure for running your scrapers
 • 
Pierluigi Vinciguerra
THE LAB #73: How to Bypass Cloudflare in 2025
Scraping websites protected by Cloudflare bot protection with open source tools
 • 
Pierluigi Vinciguerra
4
THE LAB #72: Advanced logging in Playwright
RabbitMQ, screenshots and system monitoring for your Playwright scrapers
 • 
Pierluigi Vinciguerra
THE LAB #71: Sending Scrapy logs to RabbitMQ
Saving the logs of your distributed scraping architecture to your database
 • 
Pierluigi Vinciguerra
THE LAB #70: Advanced logging in Scrapy
How to extract the most meaningful metrics from your scrapers
 • 
Pierluigi Vinciguerra
THE LAB #69: Building a dashboard for your scrapers with Grafana
Visualizing the operations of your Scrapy spider with interactive dashboards
 • 
Pierluigi Vinciguerra
THE LAB #68: Scheduling Scrapers with Airflow
How to manage a fleet of scrapers with Apache Airflow
 • 
Pierluigi Vinciguerra
1
THE LAB #67: Scraping Telegram using its APIs
How to create a bot for scraping Telegram channels
 • 
Pierluigi Vinciguerra
THE LAB #66: How to properly scrape a booking website
Business logic and best practices for scraping booking websites like Airbnb and Booking in the most efficient way
 • 
Pierluigi Vinciguerra
THE LAB #65: Scraping Datadome protected websites with Camoufox
Discovering the features of Camoufox, a custom and stealthy version of Firefox
 • 
Pierluigi Vinciguerra
2
THE LAB #64: JWT Tokens and API scraping
How to create scrapers that use token authentication for API data retrieval
 • 
Pierluigi Vinciguerra
1
THE LAB #63: Oxymouse and Playwright for human-like mouse movements
Testing the new Oxylabs open source package for human-like mouse movements
 • 
Pierluigi Vinciguerra
THE LAB #62: Bypassing Cloudflare with Nodriver
Testing the undetected-chromedriver successor for scraping Cloudflare protected websites
 • 
Pierluigi Vinciguerra
THE LAB #61: Evaluating your proxy provider
Measuring programmatically the quality of the IPs offered by proxy providers
 • 
Pierluigi Vinciguerra
THE LAB #60: Writing scrapers with LLMs
Comparing LLama3.1, GPT4 and Mistral in creating scrapers
 • 
Pierluigi Vinciguerra
The Lab #59: Bypassing certificate pinning with Frida and Fiddler - part 2
Rooting a virtual mobile Android Device to install Frida and discover API endpoints under the hood of apps
 • 
Pierluigi Vinciguerra
The Lab #58: Intercepting traffic from an App - part 1
Discover API endpoints called by an App to scrape its data
 • 
Pierluigi Vinciguerra
4
The Lab #57: Improving your Playwright scraper and avoid CDP detection
How to use the latest advancements to avoid CDP detection in your Playwright scrapers
 • 
Pierluigi Vinciguerra
3
© 2025 Pierluigi
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great culture