An ensemble model using a BabelNet enriched document space for twitter sentiment classification

Published Online January 2018 in MECS

Today, people mostly prefer to communicate and socialize on the Web. As a result of this, social media websites such as Facebook, Twitter, LinkedIn, Youtube, Instagram and so on, have exploded in popularity with tremendous impact on society in several ways. For example, in many elections around the world social media has played an important role. Many of the companies measure their customer satisfaction via social media postings; therefore, adjust the advertisement of their products to develop a better sales strategy. Apart from these, social media websites can be used to improve the people’s career and business prospects. In summary, social media is an important and major source of information to enhance the strategies range from the business and health to politics and the education system in the competitive world. Thus, to obtain the view of public from these mediums there is a need for an automated analysis system.

The sentiment classification task is extremely useful in social media analysis as it allows us to gain the common opinion behind social media messages and is broadly studied in the literature [1]. Indeed, sentiment classification is the task of learning positive or negative opinion have been expressed on certain product, organization, event or individual from opinionated documents and required in-depth analysis.

In recent years, sentiment classification from social networks has become necessary due to containing huge amount of opinionated documents. In social networks, users usually write short-texts which are also rife with emoticons, abbreviations, slangs and symbols to express their opinions. The main characteristic of a short text is the text length which is no longer than 200 characters. The sparsity and un-standardability are the two main drawbacks of short text messages. If a text sample is short, it means that the features of it are very sparse and also these limited features frequently contain non-standard terms, noise terms and abbreviations [2]. With such a text compilation, sentiment classification is becoming a more complex problem because short text examples naturally do not provide sufficient statistics or associations for classifiers. In the other hand, text classification methods generally use word statistics and associations to provide a judgment about which terms are similar or fall into which classes. In this case, short-text classification problem becomes a crucial research area in social media analysis [3].

To address the above mentioned issues, several approaches for short text classification are presented in the literature. Faguo et al. [2] proposed short text classification based on statistics and rules. Zhou et al. [4] devised a hybrid model of character-level and word-level features based on recurrent neural network with long short-term memory in order to reduce the impact of word segmentation and improve the overall performance of Chinese short text classification. Wang et al. [5] used better feature space selection to overcome sparseness problem of the short text classification. Sriram et al. [6] expanded the features of twitter texts by extracting additional author information to classify tweets with a better accuracy. For short texts in Chinese, Wensen et al. [7] used Wikipedia concepts and Word2vec to expand the features. Chen et al. [8] used LDA and K-Nearest Neighbor algorithm to improve short text classification. Sang et al. [9] expanded short texts using word embedding for classification.

Original social media messages can be preprocessed and expanded by several ways which are presented and applied as above mentioned. After these initial processes the social media messages become inputs to machine learning algorithms. Among the machine learning methods, ensemble learning allow us not only to combine multiple methods both for the preprocessing and the classification stages but also to achieve a better hypothesis. The underlying intuition of ensemble learning is to build a strong learner from a bunch of weak learner. An ensemble learning system is proposed to perform sentiment classification for twitter messages which are limited by 140 characters in length. With these kinds of short messages, language is evolving faster than ever before. In text mining systems, extracting features from text, figuring out which features are relevant, and selecting an accurate classification algorithm are fundamental questions [10].So the accurate representation of short text has become an important step of any text mining task for social media.

In this paper, the first key point of the proposed ensemble learning model is the representation of short text messages and the second one is to decide a suitable classifier. For the first key point, three subsets of the original datasets are established to construct the extended feature space. The first one is constituted only by the raw data, the second one is constituted by adding the concepts obtained from Babelnet to the raw data. The third one is constituted by the concepts obtained from Babelnet and only the extended terms by Babelnet. For the second key point, we focus on the same base learners to provide diversity which is called homogeneous classifier ensembles. The well established classification algorithms such as Naïve Bayes (NB), Support vector machines (SVM) and Decision tree (DT) are selected respectively for the weak learners of our three different ensemble learning models. The experimental results which are obtained by three public twitter datasets demonstrate that our proposed method outperforms existing classification methods in literature.

The remainder of the paper is organized as follows. In section 2, related work is summarized. In section 3, applied text preprocessing and expansion methods are defined. In section 4, methodological background of classification algorithms; in section 5, proposed ensemble model are given. In section 6 experimental results are reported and finally in section 7 we summarize the main points of our work and discuss the contributions.

• II.    Related Work

In recent years, analyzing opinion oriented human behavior has attracted the attention of researchers in artificial intelligence and natural language processing. In the literature, there are many studies on the design of methods for sentiment classification of text; however, the numbers of studies which use ensemble of classifiers are limited. Akhilesh at. al. [11] used Naive Bayes method to analyze Assembly Election on twitter users data. Indrajit at. al. [12] classified tweets into three categories, news, sport and movies by using topic modelling which is an unsupervised machine learning method. Deebha at. al. [13] formed opinion mining model according to polarity of words and examined effect of negation words. Xia et al. [14] proposed an ensemble technique for sentiment classification by integrating different feature sets and classification algorithms to boost the overall performance. Li et al. [15] used a heterogeneous ensemble learning method for Chinese sentiment classification. Su et al. [16] presented an ensemble model for sentiment classification of reviews. And their experiments showed that stacking is consistently effective compared with majority voting. Filho and Pardo [17] combined rule-based, lexicon-based and machine learning approaches for twitter sentiment classification. Hassan et al. [18] proposed the bootstrapping ensemble framework (BPEF) which provided better performance for classifying Twitter sentiments. Wang et al. [19] compared the performance of bagging, boosting, and random subspace methods based on Naive Bayes, Maximum Entropy, Decision Tree, K Nearest Neighbor, and Support Vector Machine learners for sentiment classification. They showed that random subspace has the better comperative results. Fersini et al. [20] proposed a novel ensemble method based on Bayesian Model Averaging for sentiment analysis of user reviews. They showed their model outperformed the other approaches such as bagging. Chalothorn and Ellman [21] introduced an ensemble model for twitter sentiment analysis and they formed their model by the majority voting of four base learners such as Support Vector Machine, Naive Bayes, SentiStrength and Stacking. Wang et al. [19] and Xia et al. [14] showed that the ensemble methods provide an effective way to represent short twitter texts in different ways and also improve the performance of individual base learners for sentiment classification. Cotelo et al. [22] combined the structural information and textual concepts of tweets with direct combination, stacked generalization and Multiple Pipeline Stacked Generalization methods. Isidoros Perikos and Ioannis Hatzilygeroudis [23] presented a sentiment analysis system for automatic recognition of emotions in text, using an ensemble model based on a Naïve Bayes, a Maximum Entropy learner and a knowledge-based tool. Corrêa Jr. et al. [24] proposed a tweet classification system based on ensemble of different text representations to obtain better classification performance and used Support Vector Machines and Logistic regression as base learners.

• III.    Feature Engineering

• A.    Text Preprocessing

To perform short text classification, firstly some preprocessing steps should be carried out. The most prominent characteristic of short text classification systems, such as sentiment classification in Twitter, is the shortness of the sample texts. Besides, symbols, emoticons, character repetitions, hashtags, letters in capital and abbreviations have very common usage in these kinds of texts. In this case, a lexical normalization step is needed as one of the preprocessing steps. The lexical normalization process translates nonstandard messages to Standard English and for this aim we have used a text normalization dataset generated by the Computer Science Department, The University of Texas at Dallas in this study[25]. While an original tweet in Hobbit dataset is like “my feet r killin n im dead but the hobbit was soooo good i love !!”; after the lexical normalization this tweet is become like “my foot be kill and in dead but the hobbit be so good love”. In another normalization example a sample tweet in StsGold which is like “@wale I'm gonna havta temp stop fllwing u while ur talkin abt kobe bc I loveeeeeeee him & I'm taking it personal and I like lebron 2.” is converted to “i going to have to temporary stop flow you while you talk about kobe because i love he camp am take it personal and like lebron to”.

In our study, we have removed emoticons from the tweet messages which are regarded as noisy labels as another preprocessing step. We have also removed urls, user mentions, retweet markers, as well as punctuations and digits which are not meaningful for our classification task. Then, all letters in capital were converted to lowercase letters. Consequently an input text for our classification system rifes with hashtags and preprocessed body text only. After these steps, the stemming is applied to body text by using Stanford NLP tools [26].

• B.    Concepts Generation with BabelNet

For a more accurate classification system the restricted feature space of the sample input messages are extended by using BabelNet repository which is the largest multilingual semantic network and encyclopedic dictionary composed by concepts of WordNet and Wikipedia [27]. Concepts are defined as units of knowledge, each with a unique meaning. In our proposed system, each sentence in a sample message is processed by using BabelFy API [28] and concepts from BabelNet repository are added to the obtained the feature space as new features of the related sentence. However, the stop words are kept in their original form and they are not extended by the related concepts.

For a natural language processing (NLP) system Word Sense Disambiguation (WSD) is a very challenging task in order to develop an accurate system. Expanding the term space with the term related concepts can be a way for handling ambiguity. For example, “I am taking aspirin for my cold” and “weather is cold today” are two sentences which contain “cold” term. In the first sentence the term “cold” refers to a disease, however, in the second it refers to an environmental condition. When we try to get concepts of these two sentences from BabelFy; it gives a description like “A mild viral infection involving the nose and respiratory passages” for the cold term in the first sentence and a description like “Having a low or inadequate temperature or feeling a sensation of coldness or having been made cold” for the cold term in the second sentence. Besides, term related concepts induce the increase in classifier accuracy due to generating joint feature space for terms which have common senses.

• C.    Feature Weighting

Term frequency-inverse document frequency (tf-idf) weighting scheme, which is very commonly used in text mining applications, is used to exploit the distinctiveness or importance value of a term in a document collection [29]. While tf computes the frequency of each term in a specific document, idf computes the the importance of a term for document collection. Here, a document can be viewed as a matrix, D ij = | d ij | mxn , n denotes the number documents and where m denotes the number of different of terms in the whole documents. d_i j represents the weight of term t_i in document d j . The formula of tf-idf is shown in the equation (1):

N dij = tfj x idfj = tfijx log —           (1)

Where tf  equals the number of time t appears in document d . N is the number of documents and t occurs in n documents. We use standard tf-idf to form input for machine learning algorithms, so each tweet collection is converted to a feature matrix and then classification is performed.

• IV.    Classification Algorithms

Classification is a supervised learning task, which is one of the major issues in text mining applications. The task of the classification is to assign new unobserved data to one of predefined classes by learning relationship between inputs and outputs (class labels) through available data [30]. By using available data, which is called as training data, a classification model is comprised and new unobserved data, which is called as test data, is used to validate this model. In literature there are many algorithms to perform classification in text mining area. However, to yield a better classification performance, it’s important to use a proper algorithm. In this paper NB, SVM, and DT classifiers are used due to their ability of classifying text data.

• A.    Naïve Bayes

With its simplicity and high classification accuracy NB has gained great importance and is applied to many domains including sentiment classification [31]. The assumption behind NB, which is called class conditional independence, is that all words are independent from each other within each class. The assumption of class conditional independence provides high accuracy and speed when the number of features is large. Naïve Bayes uses Bayes Theorem to compute the probability of every instance for each predefined class to assign these instances to one out of a set of the predefined classes. Based on class conditional independence, the computation of probability can be presented as the following:

n

P ( XI C i ) = П P ( ^X k lCi)         (2)

k = 1

where X={X₁,X 2 ,^,X_n} is feature vector composed of words and C1,C₂,^,C_m are class labels.

• B.    Sequential Minimal Optimization

Sequential Minimal Optimization (SMO) is one WEKA version of Support Vector Machine (SVM), breaks the training problem into quadratic programming (QP) sub-problems by using heuristics, was suggested by Platt [32]. SMO is an iterative algorithm so chooses the smallest possible sub-problem to solve at every step. The QP could be given as follows [33]:

lll max ^ ai - ^^ aiajyiyjk(XX)

i=1         i =1 j= subject to 0 < ai < c, i = 1,...,l(4)

l

£ «iVi = 0(5)

i = 1

In the Equation (3), k(X_i X j ) is the kernel function and α i is the Lagrange multiplier; in the Equation (4), c is the regularization parameter. Equations above should satisfy Karush-Kuhn-Tucker (KKT) conditions in the Equation (6) and Q ij =y i y j k(X i X j ) is positive semi-definite.

^a i  ⁰ ^ yf ⁽ X i ) ^ ¹

0 <  ^a i < ^c ^ yf ⁽ Xi ) > 1          ⁽⁶⁾

« i = ^c ^ y i f ⁽ X i ) < ¹

Due to easy implementation, be faster and better scaling properties SMO has gained great attention for sentiment classification problems [34].

• C.    Decision Tree

Based on divide and conquer approach decision tree is defined as tree structured data structure [35]. There are different algorithms for generating decision tree such as C4.5, ID3, J48, AD Tree, BF Tree, CART and many more. In this study, Quinlan’s C4.5 implementation J48 is preferred which is provided by the open-source software WEKA. Information gain calculated to select the best splitting attribute. J48 decision tree supports a variety of input data types for example nominal, textual and numeric and yields higher performance for sentiment classification tasks [36].

• V.    Ensemble Learning System

Machine learning methods have been extensively used in sentiment analysis due to their predictive performance. While the ordinary machine learning approaches that try to learn one hypothesis from the training data, ensemble methods try to construct a set of hypotheses and combine them for prediction. Ensemble learning uses a set of models which are called base learners in order to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone [19].

A labeled dataset for training, the base inducer which forms a classifier, the diversity generator which is responsible for generating diverse classifiers, and the combiner which is responsible for combining the results of the base classifiers are four building blocks that constitute a typical ensemble model [37]. Further to that, to achieve an optimal ensemble model, accuracy and diversity are two vital requirements [38]. The more diverse the base learners are, more accurate ensemble model is obtained due to diversified classifiers lead to uncorrelated classifications and improve the classification accuracy [39,40].

Methods for building ensembles can be divided into two main dimensions: how classifiers are combined and how the learning process is done [39] As for the first dimension, when combining classifiers with weights, a fixed or dynamically determined weight is assigned a learner. The majority voting is the most popular weighting combining method for base learners. In this method, a new instance is labeled due to the most frequent vote obtained from base learners [37]. Further, in weighted combination, the predictions of base classifiers are linearly aggregated. When combining classifiers with meta learning techniques, the predictions of classifiers are used as features of a meta learning model. According to [14] in trained methods, the weighted combination shows a slight superiority when compared to meta-classifiers and also simpler for sentiment classification task. Therefore we used majority voting model instead of a meta learning model our experiments.

The majority voting model combines the predictions from different classifiers using a simple fixed rule which counts the predictions of base classifiers and assigns the input to the most predicted class by classifiers. By this rule, with an ensemble model which has k=1,…,K base classifiers, the class label of the jth data point is determined as in the equation (7) [43].

K

C j = 2 I (arg ^max( C kj ) = j )        ⁽⁷⁾

k = 1

In the literature ensemble learning methods can be divided into two categories: instance partitioning methods such as Bagging and Boosting and feature partitioning methods such as Random Subspace [42]. There is not a global criteria to select a certain ensemble technique for improving the performance of a sentiment analysis task [43]. Thus, as for the second dimension, the learning process is realized with Bagging method and the well known classification algorithms are selected as weak learners of the ensemble model in this study.

In the sentiment classification of short texts task, the main issue for improvement of an ensemble model accuracy is how diversified the input text are represented rather than which classifiers are chosen as base learners. Therefore in this work, three different homogeneous classifier ensembles are formed and the well known classifiers such as NB, SVM and DT are applied as homogeneous classifiers respectively in these models. The proposed ensemble model is shown in figure 1.

Fig.1. Proposed Ensemble Model

As it is seen from figure 1, three new training datasets for three base learners are generated from original input data. The original data form (raw data) is kept as the training data of first base learner. As for the second one, the raw data and the concept words obtained from Babelnet are jointly used. As for the third one, the concepts words obtained from babelnet and only the terms which are expanded by BabelNet are jointly used. In the data of third base learner, the terms which are not considered by Babelnet are ignored. Thus only the most important terms and their concept extensions are considered as a feature space. Eventually by these base learners, the diversity of an ensemble model has been provided. As the next phase, the decided classification algorithms are used as base learners. And finally, to obtain the class label for an input message, classification outputs of three homogeneous classifiers are combined by majority voting method.

• VI.    Experimental Results

• A.    Datasets

Table 1. Description of Datasets

Dataset	#Positive	#Negative	Theme
Hobbit	354	168	Movie
IPhone6	371	161	Smartphone
UMICH	3995	3091	Movie
Archeage	724	994	Game
StsGold	596	739	General
Ststest	183	177	General

We have employed six datasets which are widely evaluated in similar studies to assess the performance of the proposed ensemble model. The whole datasets are in English, publicly available, non-encoded and involve tweets which are collected through Twitter API and labeled as negative or positive by using a Labeling tool. Table 1 summarizes the positive and negative tweet counts and theme of each datasets.

• B.    The Results Evaluation

  As the first stage of the proposed ensemble model, we have generated three training sets to provide the diversity. The three columns of table2 explain the feature spaces of these training sets respectively. The hobbit dataset originally constituted by 954 terms and this original form is used as the first training dataset of proposed ensemble model. 2182 terms which are obtained by 954 original terms and the related concepts are used as the features of second training dataset of model. By eliminating non extended terms of the original dataset 1992 features are used as the features of third dataset of model. With these three feature sets, diversity for the proposed ensemble model is provided.

Table 2. Feature Extensions

Feature Space

	Original	All terms+Concepts	Expanded terms+ Concepts
Hobbit	954	2182	1992
Iphone6	1085	2616	2414
Archeage	1963	4120	3848
UMICH	1650	3615	3387
StsGold	2522	5346	4975
StsTest	1027	2516	2372

After the comprising feature sets by extracting the semantically meaningful terms from twitter messages the model selection for training process must be realized. In this step, the well known classification algorithms such as NB, SVM and C4.5 are selected as weak learners of the ensemble model. In NB, SVM, C4.5 and ensemble experiments 80% of input data is used as training set and the remaining 20% is used as test set. The f-measure results using 10-fold cross validation are reported as in table 3.

Table 3. Classification results

Classification Methods

	NB	SVM	C4.5	ENSEMBLE
HOBBIT	0,824	0,834	0,83	0,916
IPHONE6	0,772	0,774	0,696	0,839
UMICH	0,960	0,991	0,991	0.990
ARCHEAGE	0,751	0,825	0,775	0,837
STSGOLD	0,710	0,785	0,714	0,791
STSTEST	0,690	0,617	0,709	0,717

Bold values in table 3 indicate the best classification performances. When we examine the results, it is clearly realized that the proposed ensemble model presents an overall outstanding performance to any of the other used individual classifiers. Only with UMICH dataset, the ensemble cannot increase the performances of the base classifiers. The accuracy of base classifiers for this dataset is already high such as 0.991 or 0.96. Therefore the classifiers actually cannot be considered as weak classifiers for this dataset.

When the datasets are examined in terms of theme, it can be clearly seen Sts-Gold and Sts-Test represent several domains. This causes a decline in classification performance because of domain-dependent sentiment words. Polarity of these words such as “short” change from domain to domain. When we mention about “battery life” of a smartphone for the review sentence “The battery life is very short”, sentiment word “short” has negative polarity. “The response time of the game is very short.” is about “response time” of a game and “short” has positive polarity for this review sentence. The effect of domain dependent sentiment words comes out in the classification results in Table 3. For Sts-Gold and Sts-Test the proposed ensemble model cannot increase the classification results as much as it does in the remaining datasets.

While an ensemble model compared to any other classification models, ensemble quite likely provides a superior performance due to combination of weak classifiers. The main drawback of an ensemble model is being time consuming. However if the time consuming tasks such as model selection and training can be preformed offline then the ensemble model is highly recommended due to its superior accuracy results.

• VII. Conclusion

In this paper, we have proposed an ensemble model to classify English Twitter messages according to sentiments by using word embedding representation. We indicated the need for word embedding like methods for short text classification problems; since the obtained results showed that providing word embedding by babelnet is an adequate way to represent main concepts related with the terms. The embedded knowledge extracted from Babelnet contributed the overall performance a short text classification system. Another main contribution of this study is that we propose a more accurate classifier by applying an ensemble model which includes the sub datasets which are constituted with not only the terms in original twits but also the concepts of these terms. We compared our ensemble model with widely used classification algorithms such as NB, SVM and C4.5. The comperative results which are obtained from six public datasets confirm that our ensemble model provides a more accurate classifier for short text messages.