If you are in the scraping industry, you know that you do not scrape data just for the sake of scraping. Data is very valuable, and the reason why companies need it is that they extract insights from it.
Today, machine learning, deep learning, and AI give data professionals more than just one option to provide insights from data.
In this context, I wanted to write an article that focuses on sentiment analysis on scraped product reviews. To make it more readable on Substack, I have subdivided it into two pieces:
Part #1: Discusses the theory you need to know about sentiment analysis (and that’s in the article you’re reading now).
Part #2: Provides code implementation that shows how to scrape the data from Amazon and how to perform a sentiment analysis (and this article will be published next Thursday).
Before proceeding, let me thank Decodo, the platinum partner of the month. They are currently running a 50% off promo on our residential proxies using the code RESI50.
The different methods you can use for sentiment analysis.
Reasons why to scrape Amazon product reviews using commercially available solutions (instead of writing custom code).
Ready? Let’s dive into it!
What is Sentiment Analysis and Why Do it
Sentiment analysis is a branch of Natural Language Processing focused on identifying and categorizing opinions expressed in text. It works by analyzing words, phrases, and context to determine whether the sentiment is positive, negative, or neutral. Some systems go even further, detecting emotions like anger, joy, or surprise.
The technology behind sentiment analysis ranges from keyword-based approaches to sophisticated deep learning models that understand context and nuance in text. For developers, sentiment analysis automates the extraction of subjective information from large datasets. This allows processing and interpreting feedback at scale.
The value of sentiment analysis lies in its ability to turn qualitative feedback into quantitative data. Generally speaking, it’s used to monitor public sentiment, track brand health, and inform business strategy. When applied to product reviews, sentiment analysis can become particularly impactful. Reviews are a goldmine of customer opinions, but their unstructured nature makes them hard to analyze manually. By automating sentiment detection, you can quickly assess overall product perception, identify strengths and weaknesses, and respond to customer needs effectively.
This episode is brought to you by our Gold Partners. Be sure to have a look at the Club Deals page to discover their generous offers available for the TWSC readers.
Now that you know what sentiment analysis is, let’s discuss the methodologies you can use to do it.
1. Rule-based (Lexicon-based) Methods
Rule-based sentiment analysis is the classical approach to sentiment analysis. This method uses a predefined lexicon. This is a wide list of words, each tagged as positive, negative, or neutral. The process works as follows:
Text preprocessing: It involves removing punctuation, converting to lowercase, and sometimes stripping out stopwords (common words like "the," "is," "and" that don’t add meaning).
Tokenization: Uses a tokenizer to split the cleaned text into tokens. A token is typically a word, but, depending on the tokenizer, it could also be a phrase, sentence, or even a subword. For example, the sentence “This coffee is amazing!” could be tokenized into [“this”, “coffee”, “is”, “amazing”].
Lexicon lookup: For each token, the model checks if it exists in your sentiment lexicon (the list of words with assigned sentiment scores). For example, “amazing” might have a score of +2, while “terrible” could be -2.
Scoring: Sums the sentiment scores of all tokens in the text. If a text contains both positive and negative words, it adds their scores together to get an overall value.
Handling modifiers: Some systems go further by detecting modifiers. For example, negations like “not good” flip the sentiment of “good” from positive to negative. Intensifiers like “very” or “extremely” can amplify the sentiment, so “very bad” would be scored more negatively than just “bad.”
Final sentiment assignment: Based on the total score, the model classifies the text as positive, negative, or neutral. Some systems use thresholds (e.g., score > 1 is positive, < -1 is negative, in between is neutral).
These tools are great for quick analyses and, in the context of product reviews, work well on straightforward reviews. However, they’re not context-aware—sarcasm, domain-specific jargon, or sentiment shifts can confuse them. Still, if you’re scraping tons of product reviews and want a fast result, lexicon-based methods are a good starting point. Just remember: they’re only as good as their word lists, so if your data is full of slang or niche terms, you might need to expand the lexicon or use another method.
2. Machine Learning-based Methods
Machine learning-based sentiment analysis goes a step beyond. It lets algorithms learn from labeled data. In the specific context of product reviews, the process works as follows:
Collect and label data: You start by gathering a dataset of product reviews, where each review is already labeled with its sentiment (positive, negative, or neutral). If you can not find one, you need to do this manually.
Text preprocessing: Removes punctuation and filters out stopwords. This step helps reduce noise and standardize the input.
Tokenization: Splits the cleaned text into tokens.
Feature extraction: Converts the tokens into numerical features that a machine learning model can understand. Common techniques include:
Bag-of-words: Represents each review by the frequency of each word in the text, ignoring word order.
TF-IDF (Term Frequency-Inverse Document Frequency): Weighs words based on how important they are to a specific review compared to the entire dataset.
Model selection and training: You feed the numerical features into a machine learning algorithm to train it. The model learns to associate certain patterns of features with specific sentiment labels.
Model evaluation: You test the trained model on a separate set of labeled reviews to measure its accuracy and ensure it generalizes well to new data.
Prediction: Once trained and validated, you use the model to predict the sentiment of new, unseen reviews scraped from the web.
This approach is more flexible than rule-based methods and can adapt to specific industries if you have enough labeled data.
Some parts of this process are manual, while some can be automated. The main challenge? You need a good chunk of labeled data, plus the model can’t “understand” context beyond the features you give it. Still, this can be a good compromise between simplicity and power.
Before continuing with the article, I wanted to let you know that I've started my community in Circle. It’s a place where we can share our experiences and knowledge, and it’s included in your subscription. Enter the TWSC community at this link.
Deep learning changed the sentiment analysis game, especially for complex and nuanced text. Instead of relying on hand-crafted features, deep learning models learn representations directly from raw text. You can use architectures like recurrent neural networks (RNNs), LSTMs (Long Short-Term Memory networks), and GRUs (Gated Recurrent Units), which are designed to handle sequences (texts where the order of words matters). More recently, transformer-based models like BERT and RoBERTa have become the go-to tools. These models use attention mechanisms to capture context from the entire sentence, not just nearby words.
To use these models, you typically tokenize your text with a special tokenizer (e.g., WordPiece for BERT), which splits text into subword units and maps them to IDs. Then, the model processes the tokens to output a sentiment score or class.
Deep learning models can pick up on sarcasm, idioms, and subtle cues that simpler models miss. The trade-off? They need lots of data, require a lot of computational power, and are more of a black box. For practical use, libraries like Hugging Face Transformers make it much easier to fine-tune or use pre-trained models for sentiment analysis. If your scraped data is messy or full of nuances, deep learning is worth the investment.
Hybrid sentiment analysis methods combine the best of both worlds: the speed and simplicity of rule-based systems with the adaptability of machine learning. For example, you might use a lexicon-based approach to quickly flag obvious positive or negative reviews. Then, pass the ambiguous or mixed ones to a machine learning or deep learning model for deeper analysis. Some pipelines use rule-based filters to preprocess or enrich the data before feeding it into a classifier, or they blend the outputs of both systems for a final score.
This approach is useful when you’re scraping data from multiple sources with varying language styles. You can quickly handle the “easy” cases and reserve your heavy-duty models for the tricky ones, saving compute and time.
5. API/Cloud-based Services:
If you want to skip the model-building and jump straight to results, API/cloud-based sentiment analysis services are your friend. Providers like Google Cloud Natural Language and Amazon Comprehend offer production-ready sentiment analysis via REST APIs. You send your scraped review text, and the service returns sentiment scores.
These APIs are built on top of advanced, constantly updated models, and are trained on massive, diverse datasets. They handle multiple languages, slang, and even sarcasm.
The main perks? You don’t need to manage infrastructure, scale, or retrain models. The downsides? You’re locked into their pricing, and you have less control over the model’s behavior or explainability. For most web scraping projects, these APIs are a fast and reliable way to evaluate sentiment analysis. You only need to be mindful of data privacy and API rate limits if you’re scraping at scale.
Until now, sentiment analysis should sound straightforward, right? Good! So, let’s consider some examples focused on product reviews. Say a product has the following review:
“I absolutely love this coffee maker! It’s easy to use, brews quickly, and the coffee tastes great every time.” A sentiment analysis model would classify this review as positive due to the presence of strong positive words such as “love,” “easy to use,” and “great.” The overall tone, indeed, reflects high satisfaction with the product.
Ok, great! But that was an easy case. Now consider the following case:
"The design of this backpack is fantastic. It's sleek and has tons of smart pockets. I love the look. However, the straps are incredibly uncomfortable. After wearing it for 20 minutes, my shoulders were aching. It's a major flaw in an otherwise great product.”
This is a classic case of mixed sentiment, and how it's handled depends on the model's capabilities. A simple model might output it with a "neutral" or slightly negative score, as the positive words ("fantastic," "sleek," "love the look," "great") are counteracted by the negative ones ("uncomfortable," "aching," "major flaw"). A more advanced one, instead, could provide a far more useful output. For example, some models could break the review down by topic, reporting:
Design/appearance: Positive
Comfort/straps: Negative
As a last example, consider the following:
"Oh, brilliant! The new software update made my phone's battery drain in two hours. That's a fantastic feature. I just love being tethered to a wall socket all day. Truly innovative work, guys.”
This review is a great example of why modern sentiment analysis needs to understand context. A keyword-based model would see "brilliant," "fantastic feature," "love," and "innovative" and incorrectly classify the review as positive. However, more sophisticated models learn to recognize sarcasm by identifying the conflict between overly positive words and a clearly negative event ("battery drain in two hours"). This contextual understanding allows them to override the surface-level keywords and correctly classify the sentiment as highly negative.
So, as you can understand, the nuances of this field are a lot. But they are also intriguing!
As a final proof of the importance of sentiment analysis, note that, recently, several websites have implemented a feature that summarizes the overall sentiment of all the reviews. For example, the following image shows the overall sentiment on the book “AI Engineering: Building Applications with Foundation Models” on Amazon:
Why Scraping Amazon Products Using Commercially Available Solutions
The entry point of sentiment analysis on Amazon product reviews remains scraping a target product. If you’ve been scraping for a while, you know the difficulties behind doing it on your own:
Handling pagination and login: Amazon displays customer reviews spanning several pages. To capture all data, your script must adapt to navigating through all the pages. Furthermore, to access all the reviews after the first page, you need to log in to your Amazon account. So, you also need to write a login section in your script.
Overcome blocks: Amazon blocks or temporarily suspends activities it flags as automated or robotic. So, your scraper must appear as human-like as possible. This means your script must manage fingerprinting, user-agents, proxies, and other technicalities.
HTML structure may change: You know the sensation of a scraper that breaks because a class has changed name. Amazon is no exception to that. To overcome this, you may need to consider custom scripts that go beyond only the DOM.
Accessing multiple products’ data: A common goal for performing sentiment analysis is to have more insight for comparing products. You don’t just want to compare products for prices and features: you also want to know the common sentiment. This means that you will hardly ever scrape only the reviews of a single product. So, in a scenario where you need to retrieve the data of multiple products, all with their reviews, the hurdle of scraping manually is very hard.
Commercially available solutions overcome all of these difficulties, allowing you to scrape Amazon products via API calls. This helps you easily retrieve the data at a little cost so that you can focus on what matters: products’ analysis. Common services you can use to scrape Amazon are:
Bright Data: It provides several solutions for scraping Amazon products. The most useful for the use cases you are interested in are:
Amazon scraper: It extracts data such as ASIN, seller name, merchant ID, title, URL, product overview, description, product ratings and reviews, and much more. You only need to pass it the target URLs of your products, and it provides you with a comprehensive suite of products’ data in a matter of a few seconds. This is useful if you want a wide overview of each target product to make a complete comparison.
Amazon reviews scraper: This focuses on gathering product reviews, including ratings, review descriptions, and other valuable insights. This is useful if you only want data focused on reviews.
If you use one of those services, you’ll see how easy it is to retrieve Amazon products data. Also, the fact that they provide more than one API reinforces the thesis discussed above: you are generally interested in all the products’ data. This is because you need wider information to make an exhaustive comparison among products. So, sentiment analysis is not an end goal. It is a tool to widen your analysis.
The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Conclusion
In this article, we’ve discussed sentiment analysis. You learned what it is, why perform it, and common practices.
As anticipated, this is the theoretical basis that serves as an overview of the topic. In a few days, I’ll publish another article that shows:
How to use Bright Data APIs to scrape Amazon products.
How to perform sentiment analysis on scraped Amazon products, with different techniques.
