If you have ever slowed down to reflect on your career, you know how beneficial this can be for your own path. Reflecting on how things have gone in the last years and having a somewhat clear idea of what the future will be creates your north star you can follow for your growth.
In this article, I wanted to discuss how things have changed in the scraping industry in the last 20 years or so, and how professionals have moved from scraping gigs to developing scraping products, providing consultancy, and coding scrapers for the most established firms in the industry. And yes: the idea is to also guide newbies in this industry with ideas to develop their career—hopefully, not scaring them with all the tech challenges they need to learn to deal with.
Let’s dive in!
Before proceeding, let me thank Decodo, the platinum partner of the month, and their Scraping API.
Scraping made simple - try Decodo’s All-In-One Scraping API free for 7 days.
Let’s talk about monetization first. In the early days, it was common to work on data retrieval gigs as a freelancer—either as a professional one or as a side hustler.
Essentially, the monetization evolution has shifted from mainly relying on “pay-per-gig” platforms like Upwork to selling code, datasets, and consultancy services. If you have enough experience, the monetization possibilities today are wider than in the early 2000s, thanks to the following offer possibilities:
Code as a product: There are a lot of companies out there that would rely on custom scrapers and that are willing to pay really good money for that. So, you can sell pre-built scrapers in marketplaces like the Apify store, or you can sell them directly to your clients as custom solutions. Also, note that with products like Lovable, it takes a very low effort today to create a custom micro SaaS that extracts data and lets you monetize with those customers who do not have the right technological and technical skills to manage code.
Data as a product: The point of scraping from the internet is to retrieve data from it. This means that there are companies interested only in buying scraped datasets, not your code. This is the core business, for example, of companies like Databoutique. They scrape web pages to create datasets they will sell to companies or individuals so that they can analyze them and take data-driven decisions. And yes, updating the datasets and reselling fresh data is a way of creating a retainer if you are a freelancer.
Consultancy: Scraping the web involves solving lots of technical challenges, especially today. So, if you are skilled enough, a way to monetize your expertise is by providing consultancy. You would not believe how many companies and professionals need technical consultation sessions to retrieve the data they need from the web.
Write code for companies: An important evolution of the scraping industry is that today there are a lot of companies that develop scraping solution—it was not like that at the beginning of the new millennium. Since the industry is very competitive, a good way to monetize your coding skills is to get hired by a scraping company. Note that the positions today cover a very wide spectrum of activities, ranging from being a developer to a tech leader or a CTO. The choice is only yours, based on your expectations and motivation.
So, whether you are a newbie or a well-established professional, the ways you can earn money in the web scraping field are more mature than ever since the early days of the Internet. In other words, if you are just starting, this is still a good time to do so!
This episode is brought to you by our Gold Partners. Be sure to have a look at the Club Deals page to discover their generous offers available for the TWSC readers.
It's been a while since web pages are not just static anymore. But for professional web scrapers, the challenge today does not lie only in JavaScript-heavy pages. Websites’ defences, like anti-bot systems, are more sophisticated every day, and AI is starting to get into the game. Maybe you have not focused enough on how the technical skills needed today are not only about knowing how to write code to just retrieve data. You also have to manage the following:
Proxies: There are very few occasions where you can successfully scrape a website without using a proxy. But the challenge is not only about knowing how to integrate a proxy, or a list of proxies, into a script. Over the years, proxy providers have developed several types of proxies—from residential to mobile—and knowing which one to use for your specific project will let you save a lot of money (and successfully complete a scraping project).
Browser automation frameworks: Tools like Playwright, Puppeteer, and Selenium were initially designed for automated website testing. But as new scraping challenges have emerged over the years, they have become the new standard for web scraping to control a real browser programmatically, rendering pages completely, executing JavaScript, and interacting with elements as a human would. So, let’s get this straight: today you have no possibility to survive the majority of the scraping project if you don’t know how to use at least one of them.
The anti-bot tools race: As scraping became more sophisticated, so did the defenses. Websites now employ advanced bot detection techniques, including browser fingerprinting, behavioral analysis, and sophisticated CAPTCHAs.
Anti-detect browsers: This is the other side of the coin to the anti-bot race. As bot detection got smarter by fingerprinting browsers, scrapers evolved, and anti-detect browsers are the direct answer to this challenge. They are specifically designed to create and manage separate browsing environments, each with a unique and consistent digital fingerprint. This allows your automation scripts to operate as a crowd of seemingly unrelated human users, neutralizing the tracking methods websites now rely on.
Data aggregation: Back in the days, website pages were fewer than today. Today, the Internet generates tons of new data daily. So, as a web scraping professional, you may need to aggregate data to make it more complete.
Automation and infrastructure capabilities: When coding scrapers that will be sold to customers, you may be involved in managing the infrastructure. This means you need to grow the ability to integrate a scraper with automation tools like n8n or to package it with Docker so that customers have a complete scraping pipeline that involves scraping and data integration.
Hopefully, this is not where nebies get scared with all of the stuff they need to know. If you are entering the industry now, remember that it takes years to develop a career. As I said before, the time is more mature than ever, so do not get scared by all the tech you are missing. Just start out, and remember: our newsletter is your north star to learn the best topics you need to know!
Before continuing with the article, I wanted to let you know that I've started my community in Circle. It’s a place where we can share our experiences and knowledge, and it’s included in your subscription. Enter the TWSC community at this link.
Job Opportunities: From Data Extraction to Intelligent Automation
The whole IT market has expanded and is completely changed in the last years, and this is good news also for web scraping professionals. This mainly happened because the Internet has become a fundamental part of our lives, both on the side of private and professional sides.
For professional web scrapers, this means that the opportunities in terms of industry are expanding. In other words, there are several fields that today are trending and in need of data extraction. I believe the trending ones are:
E-commerce: Do not make the mistake of thinking that customers would only pay you to scrape data from Amazon to make pricing comparisons. Spotting limited-edition sneaker releases or concert tickets in near real-time is a skill that can be worth thousands of dollars. The e-commerce industry is very mature today, and companies and individuals have a very wide spectrum of interests in extracting data from e-commerce websites.
Stocks, currencies, and cryptocurrency: In the last decade or so, the markets have entered our homes thanks to the internet. So, you do not see investors going to their broker to place an order. They do it via a mobile phone or a computer. So, performing trading by scraping prices across multiple exchanges or monitoring blockchain mempools for profitable transactions, is becoming very valuable for scraping professionals. In fact, while cryptos and blockchain are the emerging technologies, the reality is that a lot of banks and institutions are developing their own recommendation software that they sell to their customers. And guess what? Recommendation software needs to scrape data.
Social media: Social media is basically a complete Internet world inside the Internet itself, and it is a goldmine of data. Using the right techniques, you can scrape social media to perform sentiment analysis on scraped data, and this can be used for marketing (ie, understanding the sentiment on the price of a product), political (ie, understanding vote intentions based on reactions to posts), and more purposes. And this is only one of the applications where a web scraping professional can be hired to do their scraping job on social media.
The most significant technological shift is probably happening right now, driven by the explosion of AI and LLMs. For professional scrapers, examples of macro opportunities happening right now and future due to AI and LLMs are the following:
Scraping for AI: LLMs are insatiably hungry for high-quality, structured data. The next generation of foundation models is already being fine-tuned on specialized datasets. Web scraping professionals are the ones who can build the data pipelines to feed these models, extracting and cleaning niche data from across the web to create proprietary training sets for industries like law, medicine, and finance, or any other.
Using AI to scrape: If you are not integrating LLMs into your scraping pipelines, you are probably missing an opportunity. Here at The Web Scraping Club, we have already discussed in several articles that LLMs are not replacing professionals. However, integrating them into your scraping pipelines can save you a lot of headaches. For example, there are applications where using LLMs to go beyond the DOM will save you lots of hours of coding.
Scraping from AI: This is actually a new frontier. Lots of companies deliver their services—particularly, anything related to support and customer services—via conversational AI and chatbots. In this scenario, professionals who are developing techniques to automate interactions with LLMs, scrape their outputs, and monitor their behavior will surely be in high demand in the upcoming months.
In this article, I’ve gone through the path that professional web scrapers have undergone in the last 20 years or so. Surely, each career path is unique, and not everyone started when the internet was a place for a few users.
So, let us know: how has your career evolved through the years? What do you think the future holds for scraping professionals?
Are you new to the industry? Let us know how you are managing your career growth. And if you need advice, don’t hesitate to drop a comment!
The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
"Are you new to the industry? Let us know how you are managing your career growth"
I am new to this topic. I do work full time in another industry but will go part time in a couple of years. Web scraping is a topic I am interested in and also got hopefully the potential to make some money on the side (eg to pay for a vacation). To prepare for that I started learning Python, got some ideas about next steps but also see that there are quite a lot of different topics to look at ;)
Great summary, it's exciting to see the anti-detect browser topic gaining more visibility.
I'm one of the founders of Kameleo, which has been one of the pioneers in this space, and our product has been 100% focused on web-scraping automation use cases. Because of that, stability, robustness, and extremely fast performance are absolutely critical for us, they're not just nice-to-have features, but core requirements in our development.
If anyone has questions about anti-detect technology or wants deeper insights from a highly technical/operator perspective, feel free to reach out, happy to help or share more.
"Are you new to the industry? Let us know how you are managing your career growth"
I am new to this topic. I do work full time in another industry but will go part time in a couple of years. Web scraping is a topic I am interested in and also got hopefully the potential to make some money on the side (eg to pay for a vacation). To prepare for that I started learning Python, got some ideas about next steps but also see that there are quite a lot of different topics to look at ;)
Great summary, it's exciting to see the anti-detect browser topic gaining more visibility.
I'm one of the founders of Kameleo, which has been one of the pioneers in this space, and our product has been 100% focused on web-scraping automation use cases. Because of that, stability, robustness, and extremely fast performance are absolutely critical for us, they're not just nice-to-have features, but core requirements in our development.
If anyone has questions about anti-detect technology or wants deeper insights from a highly technical/operator perspective, feel free to reach out, happy to help or share more.