Ensuring data quality in web scraping projects

An example of modern web data quality control pipeline

Data quality is one of the critical pain points in web scraping: how do you know the fields in the output are correctly mapped to the information you’re looking for? Did you scrape the whole scope of your interest? Is data formatted in the correct way?

As web scraping projects increase their scope, answering all these questions becomes more and more diff…