Studying Phrase Vectors for Sentiment Evaluation: A Python Replica

We automated the evaluation and made the code obtainable on GitHub.

got here to me after I tried to breed the paper “Studying Phrase Vectors for Sentiment Evaluation” by Maas et al. (2011).

On the time, I used to be nonetheless in my closing 12 months of engineering college. The purpose was to breed the paper, problem the authors’ strategies, and, if potential, examine them with different phrase representations, together with LLM-based approaches.

What struck me was how easy and chic the strategy was. In a method, it jogged my memory of logistic regression in credit score scoring: easy, interpretable, and nonetheless highly effective when used accurately.

I loved studying this paper a lot that I made a decision to share what I realized from it.

I strongly advocate studying the unique paper. It can aid you perceive what’s at stake in phrase illustration, particularly methods to analyze the proximity between two phrases from each a semantic perspective and a sentiment polarity perspective, given the particular contexts during which these phrases are used.

At first, the mannequin appears easy: construct a vocabulary, study phrase vectors, incorporate sentiment info, and consider the outcomes on IMDb evaluations.

However after I began implementing it, I spotted that a number of particulars matter quite a bit: how the vocabulary is constructed, how doc vectors are represented, how the semantic goal is optimized, and the way the sentiment sign is injected into the phrase vectors.

On this article, we are going to reproduce the principle concepts of the paper utilizing Python.

We’ll first clarify the instinct behind the mannequin. Then we are going to current the construction of information used within the article, assemble the vocabulary, implement the semantic part, add the sentiment goal, and at last consider the realized representations utilizing the linear SVM classifier.

The SVM will enable us to measure the classification accuracy and examine our outcomes with these reported within the paper.

What downside does the paper clear up?

Conventional Bag of Phrases fashions are helpful for classification, however they don’t study significant relationships between phrases. For instance, the phrases fantastic and superb ought to be shut as a result of they specific comparable that means and comparable sentiment. However, fantastic and horrible could seem in comparable film evaluation contexts, however they specific reverse sentiments.

The purpose of the paper is to study phrase vectors that seize each semantic similarity and sentiment orientation.

Information construction

The dataset accommodates:

• 25,000 labeled coaching evaluations or paperwork
• 50,000 unlabeled coaching evaluations
• 25,000 labeled take a look at evaluations

The labeled evaluations are polarized:

• Unfavourable evaluations have scores from 1 to 4
• Optimistic evaluations have scores from 7 to 10

The scores are linearly mapped to the interval [0, 1], which permits the mannequin to deal with sentiment as a steady likelihood of constructive polarity.

aclImdb/
├── prepare/
│   ├── pos/    "0_10.txt"   -> evaluation #0, 10 stars, very constructive
│   │           "1_7.txt"    -> evaluation #1, 7 stars, constructive
│   ├── neg/    "10_2.txt"   -> evaluation #10, 2 stars, very unfavourable
│   │           "25_4.txt"   -> evaluation #25, 4 stars, unfavourable
│   └── unsup/  "938_0.txt"  -> evaluation #938, 0 stars, unlabeled
└── take a look at/
    ├── pos/    constructive evaluations, by no means seen throughout coaching
    └── neg/    unfavourable evaluations, by no means seen throughout coaching

We will subsequently retailer every doc in a Overview class with the next attributes: textual content, stars, label, and bucket.

After all, it doesn’t need to be a category particularly named Overview. Any object can be utilized so long as it gives at the least these attributes.

from dataclasses import dataclass
from typing import Non-compulsory

@dataclass
class Overview:
    textual content: str
    stars: int            
    label: str               
    bucket: str

Vocabulary development

The paper builds a set vocabulary by first ignoring the 50 most frequent phrases, then retaining the subsequent 5,000 most frequent tokens.

No stemming is utilized. No normal stopword elimination is used. That is vital as a result of some stopwords, particularly negations, can carry sentiment info.

Earlier than constructing this vocabulary, we first want to take a look at the uncooked information.

We seen that the evaluations usually are not totally cleaned. Some paperwork include HTML tags, so we take away them through the information loading step. We additionally take away punctuation hooked up to phrases, equivalent to ".", ",", "!", or "?".

It is a slight distinction from the unique paper. The authors preserve some non-word tokens as a result of they might assist seize sentiment. For instance, "!" or ":-)" can carry emotional info. In our implementation, we select to take away this punctuation and later consider how a lot this determination impacts the ultimate mannequin efficiency.

When working with textual content information, the subsequent query is at all times the identical:

How ought to we signify paperwork and phrases numerically?

The authors begin by accumulating all tokens from the coaching set, together with each labeled and unlabeled evaluations. We will consider this as placing all phrases from the coaching paperwork into one massive basket.

Then, to signify phrases in an area the place we will prepare a mannequin, they construct a set of phrases known as the vocabulary.

The authors construct a dictionary that maps every token, which we are going to loosely name a phrase, to its frequency. This frequency is solely the variety of occasions the token seems within the full coaching set, together with each labeled and unlabeled evaluations.

Then they choose the 5,000 most frequent phrases, after eradicating the 50 most frequent phrases.

These 5,000 phrases kind the vocabulary V.

Every phrase in V will correspond to 1 column of the illustration matrix R. The authors select to signify every phrase in a 50-dimensional house. Due to this fact, the matrix R has the next form:

$R in mathbb{R}^V$

Every column of R is the vector illustration of 1 phrase:$phi_w = Rw$

The purpose of the mannequin is to study this matrix R in order that the phrase vectors seize two issues on the identical time:

• Semantic info, that means phrases utilized in comparable contexts ought to be shut;
• Sentiment info, that means phrases carrying comparable polarity, must also be shut.

That is the central concept of the paper.

As soon as the info is loaded, cleaned, and the vocabulary is constructed, we will transfer to the development of the mannequin itself.

The primary a part of the mannequin is unsupervised. It learns semantic phrase representations from each labeled and unlabeled evaluations.

Then, the second half provides supervision through the use of the star scores to inject sentiment into the identical vector house.

Semantic part

The semantic part defines a probabilistic mannequin of a doc.

Every doc is related to a latent vector theta. This vector represents the semantic path of the doc.

Every phrase has a vector illustration $phi_w$, saved as a column of the matrix R.

The likelihood of observing a phrase w in a doc is given by a softmax mannequin:

$p(w mid theta; R, b) = frac{exp(theta^high phi_w + b_w)}{sum_{w’ in V} exp(theta^high phi_{w’} + b_{w’})}$

Intuitively, a phrase turns into seemingly when its vector $phi_w$ is nicely aligned with the doc vector theta.

MAP estimation of theta

The mannequin alternates between two steps.

First, it fixes R and b and estimates one theta vector for every doc.

Then, it fixes theta and updates R and b.

The theta vectors usually are not saved as closing parameters. They’re short-term document-specific variables used to replace the phrase representations.

To estimate the parameters of the mannequin, the authors use most chance.

The concept is straightforward: we wish to discover the parameters R and b that make the noticed paperwork as seemingly as potential below the mannequin.

Ranging from the probabilistic formulation of a doc, they introduce a MAP estimate θ̂ₖ for every doc dₖ. Then, by taking the logarithm of the chance and including regularization phrases, they receive the target perform used to study the phrase illustration matrix R and the bias vector b:

$nu |R|_F^2 + sum_{d_k in D} lambda |hat{theta}_k|_2^2 + sum_{i=1}^{N_k} log p(w_i mid hat{theta}_k; R, b)$

which is maximized with respect to R and b. The hyperparameters within the mannequin are the regularization weights (λ and ν) and the phrase vector dimensionality β.

On this step, we study the semantic illustration matrix. This matrix captures how phrases relate to one another based mostly on the contexts during which they seem.

Sentiment part

The semantic mannequin alone can study that phrases happen in comparable contexts. However this isn’t sufficient to seize sentiment.

For instance, fantastic and horrible could each happen in film evaluations, however they specific reverse opinions.

To resolve this, the paper provides a supervised sentiment goal:

$p(s = 1 mid w; R, psi) = sigma(psi^high phi_w + b_c)$

The vector ψ defines a sentiment path within the phrase vector house. Right here, solely the labelled information are used.

If a phrase vector lies on one aspect of the hyperplane, it’s thought-about constructive. If it lies on the opposite aspect, it’s thought-about unfavourable.

They mixed the sentiment goal and the sentiment half to construct the ultimate and the total goal studying:

$start{aligned} nu |R|_F^2 &+ sum_{ok=1}^ lambda |hat{theta}_k|_2^2 + sum_{i=1}^{N_k} log P(w_i mid hat{theta}_k; R, b) &+ sum_{ok=1}^ frac{1} sum_{i=1}^{N_k} log P(s_k mid w_i; R, psi, b_c) finish{aligned}$

The primary half learns semantic similarity. The second half injects sentiment info. The regularization phrases forestall the vectors from rising too massive.

|$S_k$| denotes the variety of paperwork within the dataset with the identical rounded worth of $s_k$. The weighting $frac{1}$ is launched to fight the well-known imbalance in scores current in evaluation collections.

Classification and outcomes

As soon as the phrase illustration matrix R has been realized, we will use it to construct document-level options.

The target is now to categorise every film evaluation as constructive or unfavourable.

To do that, the authors prepare a linear SVM on the 25,000 labeled coaching evaluations and consider it on the 25,000 labeled take a look at evaluations.

The vital query will not be solely whether or not the phrase vectors are significant, however whether or not they assist enhance sentiment classification.

To reply this query, we consider a number of doc representations and examine them with the outcomes reported in Desk 2 of the paper.

The one factor that modifications from one configuration to a different is the best way every evaluation is represented earlier than being handed to the classifier.

1. Bag of Phrases baseline

The primary illustration is an ordinary Bag of Phrases. Within the paper, this baseline is reported as Bag of Phrases (bnc). The notation means:

• b = binary weighting
• n = no IDF weighting
• c = cosine normalization

A evaluation or doc is represented by a vector v of measurement 5000, as a result of the vocabulary accommodates 5,000 phrases.

For every phrase j within the vocabulary:

$nu_j = start{instances} 1 & textual content{if phrase } j textual content{ seems within the evaluation} 0 & textual content{in any other case} finish{instances}$

So this illustration solely data whether or not a phrase seems at the least as soon as. It doesn’t rely what number of occasions it seems.

Then the vector is normalized by its Euclidean norm:

$nu_{bnc} = frac{nu}_2$

This offers the Bag of Phrases baseline used to coach the SVM.

This baseline is robust as a result of sentiment classification typically depends on direct lexical clues. Phrases equivalent to glorious, boring, terrible, or nice already carry helpful sentiment info.

2. Semantic-only phrase vector illustration

The second illustration makes use of the phrase vectors realized by the semantic-only mannequin.

The authors first signify a doc as a Bag of Phrases vector v. Then they compute a dense doc illustration by multiplying this vector by the realized matrix:

$z_{textual content{semantic}} = R_{textual content{semantic}} occasions nu$

The place $R_{textual content{semantic}} in mathbb{R}^{50 occasions 5000}, nu in mathbb{R}^{5000} quadimpliesquad z_{textual content{semantic}} in mathbb{R}^{50}$

This vector could be interpreted as a weighted mixture of the phrase vectors that seem within the evaluation.

Within the paper, when producing doc options by the product Rv, the authors use bnn weighting for v. This implies:

• b = binary weighting
• n = no IDF weighting
• n = no cosine normalization earlier than projection

Then, after computing Rv, they apply cosine normalization to the ultimate dense vector.

So the ultimate illustration is:

$bar{z}_{textual content{semantic}} = frac{R_{textual content{semantic}} nu}{| R_{textual content{semantic}} nu |_2}$

This illustration makes use of semantic info realized from the coaching evaluations, together with each labeled and unlabeled paperwork.

3. Full semantic + sentiment illustration

The third illustration follows the identical development, however makes use of the total matrix Rfull​.

This matrix is realized with each elements of the mannequin:

• the semantic goal, which learns contextual similarity between phrases;
• The sentiment goal, which injects polarity info from the star scores.

For every doc, we compute:

$z_{textual content{full}} = R_{textual content{full}} nu$

Then we normalize:

$bar{z}_{textual content{full}} = frac{R_{textual content{full}} nu}{| R_{textual content{full}} nu |_2}$

The instinct is that $R_{full}$ ought to produce doc options that seize each what the evaluation is about and whether or not the language is constructive or unfavourable.

That is the principle contribution of the paper: studying phrase vectors that mix semantic similarity and sentiment orientation.

4. Full illustration + Bag of Phrases

The ultimate configuration combines the realized dense illustration with the unique Bag of Phrases illustration.

We concatenate the 2 representations to acquire:

$x = left[ bar{z}_{text{full}} ;middle|; nu_{bnc} right]$

This offers the classifier two complementary sources of knowledge:

• a dense 50-dimensional illustration realized by the mannequin;
• a sparse lexical illustration that preserves actual word-presence info.

This mixture is beneficial as a result of phrase vectors can generalize throughout comparable phrases, whereas Bag of Phrases options preserve exact lexical proof.

For instance, the dense illustration could study that fantastic and superb are shut, whereas the Bag of Phrases illustration nonetheless preserves the precise presence of every phrase.

We then prepare a linear SVM on the labeled coaching set and consider it on the take a look at set.

This enables us to reply two questions.

First, do the realized phrase vectors enhance sentiment classification?

Second, does including sentiment info to the phrase vectors assist past semantic info alone?

Implementation in Python

We implement the mannequin in 5 steps:

1. Load and clear the IMDb dataset
2.  Construct the vocabulary
3. Prepare the semantic part
4. Prepare the total semantic + sentiment mannequin
5. Consider the realized representations utilizing SVM

The desk beneath reveals the closest neighbors of chosen goal phrases within the realized vector house.

For every goal phrase, we report the 5 most comparable phrases in keeping with cosine similarity. The complete mannequin, which mixes the semantic and sentiment goals, tends to retrieve phrases which can be shut each in that means and in sentiment orientation. The semantic-only mannequin captures contextual and lexical similarity, but it surely doesn’t explicitly use sentiment labels throughout coaching.

The desk beneath compares our outcomes with the outcomes reported within the paper. For every illustration, we prepare a linear SVM on the labeled coaching evaluations and report the classification accuracy on the take a look at set. This enables us to judge how nicely every doc illustration performs on the IMDb sentiment classification process.

The complete mannequin may be very near the outcome reported within the paper. This means that the sentiment goal is carried out accurately.

The biggest hole seems within the semantic-only mannequin. This may occasionally come from optimization particulars, preprocessing, or the best way document-level options are constructed for classification.

Conclusion

On this article, we reproduced the principle elements of the mannequin proposed by Maas et al. (2011).

We carried out the semantic goal, added the sentiment goal, and evaluated the realized phrase vectors on IMDb sentiment classification.

The mannequin reveals how unlabeled information may also help study semantic construction, whereas labeled information can inject sentiment info into the identical vector house.

It is a easy however highly effective concept: phrase vectors mustn’t solely seize what phrases imply, but in addition how they really feel.

Whereas this publish doesn’t cowl each element of the paper, we extremely advocate studying the authors’ unique work. Our purpose was to share the concepts that impressed us and the enjoyment we discovered each in studying the paper and scripting this publish.

We hope you get pleasure from it as a lot as we did.

Picture Credit

All photos and visualizations on this article had been created by the writer utilizing Python (pandas, matplotlib, seaborn, and plotly) and excel, except in any other case acknowledged.

References

[1] 𝗔𝗻𝗱𝗿𝗲𝘄 𝗟. 𝗠𝗮𝗮𝘀, 𝗥𝗮𝘆𝗺𝗼𝗻𝗱 𝗘. 𝗗𝗮𝗹𝘆, 𝗣𝗲𝘁𝗲𝗿 𝗧. 𝗣𝗵𝗮𝗺, 𝗗𝗮𝗻 𝗛𝘂𝗮𝗻𝗴, 𝗔𝗻𝗱𝗿𝗲𝘄 𝗬. 𝗡𝗴, 𝗮𝗻𝗱 𝗖𝗵𝗿𝗶𝘀𝘁𝗼𝗽𝗵𝗲𝗿 𝗣𝗼𝘁𝘁𝘀. 2011. Learning Word Vectors for Sentiment Analysis. In Proceedings of the forty ninth Annual Assembly of the Affiliation for Computational Linguistics: Human Language Applied sciences, pages 142–150, Portland, Oregon, USA. Affiliation for Computational Linguistics.

Dataset: IMDb Large Movie Review Dataset (CC BY 4.0).