Evaluating sentiment analysis and word embedding techniques on Brexit

ABSTRACT


INTRODUCTION
Sentiment analysis [1] is commonly used in the context of social media, as digital communication networks produce a significant amount of written content, it can be examined to discern the attitudes of those who utilize it.This can include analyzing the overall sentiment [2] of a particular brand or product or identifying sentiment towards specific topics or events.There are several challenges in applying sentiment analysis to social media data [3], including the informal and often abbreviated nature of the text, as well as the presence of slang, misspellings, and other forms of non-standard language.However, with the use of advanced natural language processing techniques [4], it is possible to accurately identify the sentiment of social media [5] posts and use this information to gain insights about the attitudes and opinions of users.
Sentiment analysis is a subfield of natural language processing that centers on utilizing machinelearning techniques [6] to recognize and extract objective information from written content.This information can include the emotional tone of the text, as well as the overall sentiment (positive, neutral, or negative) expressed by the writer.By applying sentiment analysis to large datasets of text [7], such as social media posts or customer feedback, organizations can gain insights into the opinions and emotions of their audience.
One of the main advantages of sentiment analysis is its ability to help organizations make decisions that are more informed by providing them with a deeper understanding of their customers needs and preferences.For instance, a company might use sentiment analysis to analyze customer feedback [8] and identify common trends or patterns that could be used to improve their products or services.Furthermore, sentiment analysis can be used to monitor social media platforms for mentions of a particular brand or product, allowing companies to quickly respond to customer complaints or concerns.Sentiment analysis plays a crucial role in extracting valuable insights from vast volumes of text data.Leveraging machine learning techniques [9], organizations can effectively identify and analyze the sentiment expressed within text, enabling them to make more informed decisions.Moreover, this analytical approach empowers companies to enhance their products and services based on the feedback and sentiments expressed by their customers.
Sentiment analysis frequently employs machine-learning techniques to automatically discern the attitude in written content.These approaches are educated on a vast dataset of annotated text, where the annotations indicate the emotion of the text (e.g.positive, negative, and neutral).The machine-learning method [10] utilizes this training data to learn the patterns that are connected with diverse emotions, and can then be used to new, unseen text data to predict the emotion of the text.
In sentiment analysis, a wide range of machine learning algorithms can be employed [11].These encompass traditional classification methods like support vector machines and decision trees, alongside more advanced neural networks including long short-term memory (LSTM) networks and convolutional neural networks (CNNs) [12].The selection of the most suitable algorithm hinges on factors like the dataset's unique characteristics and the performance objectives set for the sentiment analysis system.
One of the most popular methods to represent words is known as word embedding [13].Word embedding is a technique for representing words as vectors in a high-dimensional space.These word vectors capture the semantic meaning of the words, and the position of the vector in the space encodes the meaning of the word.Word embedding is a vital aspect of numerous natural languages processing assignments, including opinion mining and machine interpretation.
By using word embedding in conjunction with sentiment analysis [14], the sentiment analysis model can learn to associate specific words or phrases with certain sentiments.For example, a word-embedding model may learn to associate the word "terrible" with negative sentiment, while associating the word "wonderful" with positive sentiment.This can help the sentiment analysis model to predict the sentiment of a piece of text more accurately, even if the text contains words or phrases that the model has not seen before.
Pre-trained word embedding models serve as a valuable resource for natural language processing tasks [15], such as sentiment analysis, by utilizing their training on extensive text datasets.These models come equipped with a comprehensive understanding of semantic relationships between words, allowing them to offer meaningful word representations in the form of word vectors.As a result, they serve as a convenient starting point for sentiment analysis, enabling researchers and practitioners to leverage the pre-existing knowledge encoded within these models to enhance their sentiment analysis algorithms.
Employing pre-trained word embedding models in sentiment analysis can aid to augment the effectiveness of the emotion recognition model.Because the pre-trained word-embedding model has already learned the semantic connections between words, it can supply useful information to the sentiment analysis model regarding the significance of words and phrases in the text data.This can assist the emotion recognition model to recognize the emotion of the text more accurately.
A wide range of pre-trained word embedding models are readily accessible for various natural language processing tasks.Among the popular options are word to vec (word2vec) [16], global vectors for word representation [17], embeddings from language models (ELMo) [18], and FastText [19], each offering unique advantages and capturing different aspects of word semantics.These well-known pre-trained models have been widely adopted by researchers and practitioners to facilitate tasks like sentiment analysis, providing a solid foundation for understanding word meanings and contextual relationships within textual data.We posit that integrating pre-trained word embeddings like GloVe, word2vec, ELMo, and FastText into sentiment analysis tasks will enhance accuracy and effectiveness.These embeddings, trained on extensive datasets, have already encapsulated semantic word relationships.By incorporating them into our sentiment analysis model, we anticipate improved accuracy in identifying and classifying sentiments compared to using a basic word-embedding layer.In this study, we aim to work and compare the results of the different pretrained word embedding models on Brexit data, which are used here [20].This paper is organized as such: i) Section 2 will list all the methods used in this study in detail; ii) Section 3 will summarize the results we got and their interpretation; and iii) The last section is a round-up of the paper and will conclude the paper.

METHOD
In this research, we aim to develop various methods to compare word-embedding techniques in the field of sentiment analysis [21].We divide our architecture into 5 fundamental stages as illustrated in Figure 1.The first step is to identify an appropriate dataset [22] that can yield good results during the training of our models.Next, we proceed to preprocessing [23] with the aim of cleaning our data without damaging the accuracy of the final models.In the third stage, we commence building the various types of word vector representations [24] for our embedding layer to be used as input for the fourth stage, based on the pre-trained In the fourth stage, we create a neural network classifier [25] to train our final model that is based on the prior dataset and the vectors of the embedding layer.Finally, we apply the classifier to Brexit data, allowing us to compare it with the results of our previous study.
Figure 1.The foundational elements of our structures

Dataset
Within the scope of this study, we designed an experiment to assess different datasets with the aim of identifying the most appropriate one for our specific objectives, ultimately yielding the highest accuracy in our final model.Through the training process involving diverse datasets such as tweets, internet movie database (IMDb) reviews, Amazon reviews, and Yelp reviews, we determined that the tweets dataset emerged as the most optimal choice, delivering exceptional results that aligned closely with our requirements.Consequently, the utilization of the tweet's dataset proved to be instrumental in achieving our desired outcomes within this study.

Preprocessing
Before we start the main step of creating the models, we should apply preprocessing techniques to our dataset as described in Figure 2. We started by converting the text to lowercase.Then, we applied some regex to delete any HTML tags or links.Meanwhile, we tried to clean any specific characters such as numbers and punctuation.Moreover, before we applied lemmatization to the text, we tokenized the sentence.In this work, we tried not to complicate the preprocessing stage with other stemming and filtering techniques because it is hard to improve the final accuracy of the model when we apply many filters to one text.

Word embedding
Word embedding, or word representation, is a technique used in natural language processing (NLP) [26].Each word is represented in low-dimensional vectors based on numbers.When using word embedding, the semantic information of words can be captured from a large corpus.Word embeddings are used in different tasks of natural language processing to provide the best word representation.There are many types of word embedding algorithms, such as ELMo [27], GloVe [28], word2vec [29], and FastText [30].In this study, we work with the pre-trained models of these four techniques.

GloVe
Global vector word representation (GloVe) is based on the co-occurrence and factorization of a matrix to generate their vectors.The idea is to find the relationship between words from a statistical point of view.GloVe starts by constructing a large matrix of words x context and stores the co-occurrence information as shown in Table 1.In this study, we used a pre-trained GloVe word embedding generated by Standford University that was trained on 840 billion words, with 300 dimensions.

Word2vec
Word2vec is a word representation technique that utilizes the presence of words within written content to establish connections between words.For instance, word2vec might relate the words "females" and "males" since they often appear in similar settings.word2vec has two forms of architecture: context-prediction, which predicts the surroundings of a given word, and context-based-prediction (Bag-of-words), which predicts a word from a given surroundings.In essence, word2vec takes as input a written corpus and produces as output a word vector.In this research, we employed a pre-trained word-vectorization model that was trained on the Google News corpus, which comprises about 100 billion words and has 300-dimensional vectors.

FastText
FastText is a tool created by Facebook that is used for text classification and word representation.One of the key advantages of FastText is its ability to generate better word embeddings for rare words using n-gram character vectors.In this study, we used FastText to obtain the weights for our embedding layer based on a pretrained model that was trained on a 2-million-word vector on common crawl and has 300-dimensional vectors.

ELMo
ELMo characterizes a sequence of words as a sequence of vectors.It employs a bi-directional LSTM model to construct its word representations.Additionally, the benefit of ELMo is that a word can have various vector representations based on the context.For instance, the word "pail" in the following two sentences: "He let go of the pail," and "I have a list of things to do before I die, a pail list.The word "pail" has different meanings in both sentences.In the ELMo method, different vectors will represent the word «pail» because it is surrounded by different words, which means different contexts.This is in contrast with other methods, which will give the same vector for both situations.In this study, we used a pre-trained model of ELMo provided by Google.The parameters we used in our research are the default signature and as_dict set to true.

Building our classifier model
In this study, for GloVe, word2vec and FastText, we built a neural network that contains the following layers: the embedding layer, flatten layer, and dense layer.The activation function used is softmax.Figure 3 describes the parameters.
For the loss, we used "categorical_crossentropy" and Adam as the optimizer method, with 'accuracy' as the metric.For ELMo, we built the following layers: an embedding layer that takes text as input, a first dense layer that takes the embedding layer as input with 'relu' as the activation function, and a second dense layer with a sigmoid as the activation function.We also used 'binary_crossentropy' as the loss function, 'rmsprop' as the optimizer, and 'accuracy' as the metric.

RESULTS AND DISCUSSION
Brexit refers to the United Kingdom's exit from the European Union (EU).In 2016, the UK voted in a referendum to leave the EU.The decision to leave the EU has sparked a great deal of political debate and controversy within the UK, as well as with other countries in the EU.Some of the key issues surrounding Brexit include immigration, trade, and sovereignty.The process of leaving the EU has been complex and has involved negotiations between the UK and the EU to determine the terms of the UK's withdrawal, as well as the future relationship between the UK and the EU.
In this study, we performed sentiment analysis and word embedding on a dataset of tweets from kaggle.For the sentiment analysis, we employed an LSTM model to categorize the sentiment of each text as positive, negative, or neutral.For the word embedding, we used a pre-trained model to map each word and phrase in the dataset to a high-dimensional vector, allowing us to analyze the relationships between different words and phrases in the context of the dataset.

Accuracy of our models
Table 2 presents a summary of the accuracy of our models.The table shows the results of four different models: Glove, word2vec, ELMo, and FastText.The accuracy of each model is measured on a scale of 0 to 1, with 1 being a perfect score.The results indicate that all models performed well, with Glove and FastText achieving an accuracy of 0.88, word2vec achieving an accuracy of 0.87, and ELMo achieving an accuracy of 0.86.Overall, the table shows that all models performed similarly, and achieved high accuracy scores, which suggests that all of the models are suitable for use in sentiment analysis tasks.

Reports and metrics of our models
Tables 3 and 4 present the results of evaluating the performance of the pre-trained word embedding models, GloVe and word2vec respectively, using several different metrics.The tables show the results for precision, recall, and F1-score for each model.The precision metric measures the proportion of true positive results among all positive results, recall measures the proportion of true positive results among all actual positive observations, and F1-score is the harmonic mean of precision and recall.The tables also show the accuracy of each model, which is the proportion of correctly classified observations.The table also show macro avg and weighted avg.
The evaluation of the GloVe model reveals impressive performance metrics across multiple categories.With a precision of 0.87 for the negative class and 0.89 for the positive class, the model showcases its ability to accurately classify sentiment.Additionally, the model exhibits a recall of 0.88 for both classes, indicating its capacity to effectively capture instances of sentiment expression.Furthermore, with F1-scores of 0.88 for the negative class and 0.89 for the positive class, the GloVe model demonstrates a balanced performance in terms of precision and recall.Overall, the model achieves an accuracy of 0.88, highlighting its proficiency in sentiment analysis, as reflected in both the Macro avg and weighted avg scores, which also stand at 0.88.
Upon analyzing the performance of the word2vec model, noteworthy findings come to light.The precision of 0.87 achieved for both classes signifies the model's ability to accurately classify sentiment across the board.With a recall of 0.86 for both classes, the model demonstrates its proficiency in capturing sentiment expressions comprehensively.Furthermore, the F1-scores of 0.86 for the negative class and 0.88 for the positive class exemplify a balanced performance in terms of precision and recall.Overall, the word2vec model attains an accuracy of 0.87, as reflected in both the Macro avg and weighted avg scores, further solidifying its efficacy in sentiment analysis tasks.Based on the tables, it is evident that both GloVe and word2vec models exhibit commendable performance, showcasing comparable results across all evaluation metrics.The closely aligned precision, recall, and f1-score values signify a well-balanced nature of these models, indicating their proficiency in accurately predicting sentiment for both positive and negative classes.These findings emphasize the reliability and effectiveness of both GloVe and word2vec in sentiment analysis tasks, underscoring their capability to provide valuable insights into the sentiment expressed within textual data.

Results
After completing our analysis, we are delighted to share the results.We put in a great deal of effort to carefully evaluate the data and arrive at these conclusions.We believe that the findings of our study will provide valuable insights and help advance our understanding of sentiment analysis and pre-trained word embeddings techniques.We hope that you find the results as interesting and informative as we do.
In the Table 5, we present the results of our research study that we explained before for comparing various pre-trained word embedding models.The results include a comparison to our previous work on the Brexit topic, as well as statistics from NatCen's.We believe that these results provide valuable insights into the performance of different word embedding models and can help guide future research in this area.
Table 5 shows the results of our research study comparing various pre-trained word embedding models.The table compares the performance of Glove, word2vec, FastText, Elmo, LSTM and NatCen's, in terms of the percentage of accurately classified samples of remain in EU and leave EU, regarding the Brexit topic.The results show that Glove and word2vec are the best performer with 73.56% and 75.26% respectively, followed by FastText, ELMo, LSTM, and NatCen's with 65.48%, 61.21%, 54.88%, and 55.55% respectively.In this study, we aim to highlight the differences between using a simple word embedding layer and a pre-trained layer for sentiment analysis.The use of word embeddings in natural language processing (NLP) has shown significant improvement in various NLP tasks, including sentiment analysis.Word embeddings represent words in a low-dimensional vector space, where the distance between the vectors captures the semantic relationships between the words.To demonstrate these differences, we will provide an example of a tweet related to Brexit.Then, we will explain how a simple word embedding layer and a pre-trained layer works in a sementic perspective: "Brexit negotiations are going nowhere.It's like watching a game of chess where both sides are stuck in a stalemate." If we use a general embedding layer, it will generate word embeddings for each word in the sentence without any prior knowledge or training on a specific task.These embeddings will be based on the distributional semantics of the words, which means that words that appear in similar contexts are likely to have similar embeddings.For example, the embedding for "Brexit" and "negotiations" may be similar since they appear in the same sentence and are related to the same topic.However, a general embedding layer may not be able to capture the full semantic meaning of the sentence or the sentiment behind it.
On the other hand, if we use a pre-trained word embedding like GloVe, it has been trained on a large corpus of text and has already captured the semantic relationships between words.Therefore, it will be better at capturing the meaning of the sentence and the sentiment behind it.For example, GloVe may be able to capture the negative sentiment in the sentence and the fact that Brexit negotiations are not progressing, which may be reflected in the embeddings for "going nowhere" and "stalemate."Overall, using a pre-trained word embedding like GloVe can be more effective than a general embedding layer in capturing the semantic relationships and sentiment in a sentence.

CONCLUSION
In conclusion, our study has demonstrated the effectiveness of pre-trained word embedding models for sentiment analysis.Through a series of experiments, we were able to show that these models can achieve high levels of accuracy when applied to a variety of text data.Furthermore, our analysis of the output of the models provided valuable insights into the sentiments expressed in the data.One limitation of using pre-trained word embeddings for sentiment analysis is that they are based on a fixed set of relationships between words, which may not always be relevant or appropriate for a specific task or dataset.For example, a pre-trained word embedding model trained on a general-purpose dataset may not capture domain-specific terminology or relationships that are important for a sentiment analysis task in a specific industry.Additionally, pre-trained word embeddings may be biased due to the biases present in the dataset used to train them.This can lead to incorrect or unfair sentiment classification, particularly for texts that deal with sensitive topics or marginalized groups.Finally, pre-trained word embeddings may not be able to accurately capture the sentiment of novel or rare words that were not present in the training dataset, leading to errors in classification.Looking to the future, we believe that continued research in this area will help to further improve the performance of sentiment analysis models.In particular, the development of new and more sophisticated pre-trained word embedding models will likely play a key role in this progress.Furthermore, advances in natural language processing and machine learning algorithms will help to enable sentiment analysis models to be applied in a wider range of contexts, including new domains and languages.Overall, we are optimistic about the potential of pre-trained word embedding models to advance the field of sentiment analysis.These models offer a powerful tool for extracting sentiment information from text data, and we believe that they will continue to play a crucial role in this area of research and development.

Int
Evaluating sentiment analysis and word embedding techniques on Brexit (Ihab Moudhich) 697 model.

Figure 2 .
Figure 2. The data preparation techniques for our dataset

Int
Evaluating sentiment analysis and word embedding techniques on Brexit(Ihab Moudhich)    701

Table 1 .
Unlocking the power of word representation through matrix construction for GloVe

Table 2 .
The accuracy of our models: a summary

Table 3 .
Measuring the performance of GloVe

Table 5 .
Analyzing the performance of word embedding models: a comparative study