Analysis of Emotion Recognition Model Using Electroencephalogram (EEG) Signals Based on Stimuli Text

Article History: Received: 10 November 2020; Revised: 12 January 2021; Accepted: 27January 2021; Published online: 05April 2021 Abstract:Recognizing emotions through the brain wave approach with facial or sound expression is widely used, but few use text stimuli. Therefore, this study aims to analyze the emotion recognition experiment by stimulating sentiment-tones using EEG. The process of classifying emotions uses a random forest model approach which is compared with two models, namely Support Vector Machine and decision tree as benchmarks. The raw data used comes from the results of scrapping Twitter data. The dataset of emotional annotation was carried out manually based on four classifications, specifically: happiness, sadness, fear, and anger. The annotated dataset was tested using an Electroencephalogram (EEG) device attached to the participant's head to determine the brain waves appearing after reading the text. The results showed that the random forest model has the highest accuracy level with a rate of 98% which is slightly different from the decision tree with 88%. Meanwhile, in SVM the accuracy results are less good with a rate of 32%. Furthermore, the match level of angry emotions from the three models above during manual annotation and using the EEG device showed a high number with an average value above 90%, because reading with angry expressions is easier to perform. For this reason, this study aims to test the emotion recognition experiment by stimulating sentiment-tones using EEG. The process of classifying emotions uses a random forest model approach which is compared with two models, namely SVM and decision tree as benchmarks. The dataset used comes from the results of scrapping Twitter data.


Introduction
Emotion recognition in the computer field is growing rapidly nowadays Computer machine can identify the same emotion recognition and continue learning to the new data rapidly. Several studies using different approach to detect the human emotion aims to improve the communication amongst human and machine and to make an effective, usable, and easier interaction. Furthermore, the elaboration of NLP and Brain Computer Interface (BCI) is becoming more interesting and challenging due to the massive development of the Internet, especially social media. The research of human and machine interaction has become an important part of today.
Various algorithm models are offered in emotion recognition with a dataset of words on the Internet. Also, sentiment analysis is carried out in several studies for identification and classification purpose. However, the emotional recognition stages such as analyzing the text structure model on social media [1], classifying hate speech in tweets [2], as well as several algorithms, such as random forest [3], [4], and deep learning methods [5] has become the concern of researchers. Furthermore, pre-processing stages in text analysis was also conducted to recognize the emotional recognition. Some researchers carried out study to classify the emotions using feature extracting from the text by the word embedding technique [6]. The other research was conducted to compare several algorithms in machine learning [7], as well as a survey of sentiment analysis models to detect and recognize emotions [8]- [10].
The research on BCI has also been widely carried out, especially in emotional recognition. Consequently, [11], [12] conducted a survey in recognizing brain wave activity in various events and found that it controls thoughts, feelings, and behavior. Furthermore, human speech, behavior, and writing indicate a mood or emotion that is going on [13]. Moreover, the dataset on various stimuli in determining brain waves is also a concern for the researchers to analyze emotions [14]. The Electroencephalogram (EEG) signals that the brain's electrical information has been highly valued over the past few years. In addition, an EEG device, depending on the application, uses 3 to 256 electrodes placed on the scalp to record relative voltages. For example, in clinical applications, 8 to 32 EEG channels are usually required.
However, few findings use EEG to translate emotions in texts with a little difference in the process of analyzing the sentiment. Thus, the main objective of this research is to analyze several models for emotion recognition for detecting the suitability level of reader's emotions based on texts stimuli modal using EEG signals.

Related Works
Several findings of previous related articles have provided an important guide for this research. Consequently, the articles related to this research are the process of extracting features from signal waves to numbers, performing text sentiment annotations for stimuli, and algorithmic models for classifying emotions.
An article by [15] found new extraction features to get better accuracy valuesin detecting emotions by dividing two features, namely time and frequency domain. In this research, the classifiers used are SVM, KNN, and neural network. Furthermore, another article uses the extraction method in building vector features by also comparing three classifiers, namely rule-based, decision tree, and SVM [16].
Besides, there is a survey article about several methods of the framework and components of BCI, band, and frequency of brain wave signals that was carried out by (Deepak et al. 2015) and a review paper on several types of the systems to recognize human emotions [12]. As a result, these articles give ease to determine the model with the best accuracy. In the table above, the algorithm widely used to recognize emotions with Electroencephalogram signals are naïve Bayes, SVM, decision tree, and machine learning techniques as well as random forest models. Furthermore, the stimuli modal for the dataset on brain waves mostly use facial and speech (audio) expressions. Meanwhile, detecting emotions with text-based is still rarely studied.

Research Methodology
The proposed method for emotion recognition analysis consists of the following tasks, namely dataset collection, initial processing, signal extraction, and algorithm model analysis. Figure 1 shows each step in the proposed method.

Figure 1. The Conceptual Framework of Emotion Recognition Model
The Electroencephalogram signals and sentiment texts from twitter crawled 6000 tweet dataset. However, two participants annotated the it to four basic emotion classification. EEG signals were recorded using an Emotiv EPOC wireless device, but videos are not used in this research. During the collection of data, stimulants were presented to each participant in the order of happy, sadness, fear, and angry emotions.

Data Collection
In the first step, we compiled twitter tweets with the topic of the 2019 presidential election, which was collected from January 1, 2018, to October 9, 2018 by scrapping technique, with related emotions that were manually annotated. Scrapping is a process of getting small fragments of something. In our case, it is web scraping, so here we are taking fragments of information available on a website. This study, the process of retrieving of the twitter data scraping using the "tweet-scraper" module found in Python, with the keyword "2019 presidential election". Scraping twitter data was taken from January 1, 2018, to October 9, 2018. Data obtained as many as 116,961 tweets containing "presidential election 2019". The raw from crawling data twitter obtained is stored in JSON format.
Below are the tweets result after being annotated manually by the participants. Besides, the classification process grouped the same data of 1000 tweets into four classes of emotion words, which include happy, sad, angry, and fear. Consequently, each document classified will be categorized into the most dominant class.

Data Acquisition
The brain's electrical activity recorded using the EPOC headset manufactured by Emotiv EPOC neuroheadset, a 14-channel (plus CMS/DRL references, P3/P4 locations) high-resolution wireless neuroheadset with neuro signal acquisition and processing function (Emotiv 2014). The available electrode positions can be seen in Figure 3. The experimental setup for data acquisition process consists of several devices including emotiveEPOC+, a presentation screen, signal analysis software, and twitter data collection. Consequently, the dataset above was annotated again based on EEG signal, using Emotive EPOC 14+ which contains 14 channels brainwave sensor, namely AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4. TheF3 and F4 sensor was used for neuroscan, and wasrecorded at 1000 Hz, meanwhile AF3 and AF4 wasused for EPOC. Besides, EEG device was placed in the user's head with appropriate electrode in sensors. While the Emotiv EPOC uses a sequential sampling method at a rate of 128 samples per seconds (SPS).

Participants
Two participants were invited to capture the brain wave while reading the sentiment text using the EEG tool (EmoticEpoc+). Electroencephalogram data were recorded using a device for two healthy subjects, and both are male. The two participants were students from Bandung, West Java Indonesia, whose age ranges from 18 years. The text was read carefully so that it can be understood and the intonation was adjusted to the label. The following of procedure for collecting data by participants is: 1) Read the text for 2 minutes; 2) Close eyes to arouse emotions for 1 minute; 3) Open eyes for 1 minute; and 4) Reread the same text with emotions that have built for 4 minutes Material.

Result and Discussion
The stimuli used are a set of text-based posts/comments taken from Twitter, which are the same as the data in emotion classification with sentiment analysis. The brain signal was recorded in 1-2 hours speech of a user while reading political domain tweets. However, the signal changed into 14 class numerical EEG brain representations known as features. Consequently, after reading 63200 tweets, 6300 features vector of reading political domain were produced.

EEG Signal Processing
The signal-processing block involves the pre-processing, feature extraction, and classification steps. Furthermore, the EEG signal pre-processing involves a variety of techniques which were applied to reduce noise and remove artifacts so that a clean signal is ready for the next step. Therefore, several stages were done such as the conversion of brain wave data into numbers. Consequently, the whole of the signal were covered, even though there were many unnecessary data. However, unnecessary columns which can be truncated using Python programming are deleted. Furthermore, feature selection was used to remove misleading data and noise, thereby resulting in increased accuracy. After collecting the 63200 EEG feature vector, the data was pre-processed in order to produce an appropriate value. The feature is scaled into the range between 0 -1 and the value below the threshold known as idle condition were removed. However, while reading the tweet, several steps from raw data used to classify emotion detection were also followed. Figure 5. The extraction feature of brain wave data into numbers The stimuli used are a set of text-based posts/comments taken from Twitter, which are the same as the data in emotion classification with sentiment analysis. The brain signal was recorded in 1-2 hours speech of a user while reading political domain tweets. However, the signal changed into 14 class numerical EEG brain representations known as features. Consequently, after reading 63200 tweets, 6300 features vector of reading political domain, were produced.
Meanwhile, the Librosa tool in the python package was used in this experiment for feature extraction. Also, in the EEG Emotiv EPOC+ type, a software capable of converting waves into numerical (.csv) forms were gotten, so as to make a feature selection to obtain a clean data. The following are the figure of instance of visualization signal for each emotion texts and the result of sum of the sentiment frequencies from EEG feature extraction and selection, where from 63200 texts to 19990 sadness texts, 17546 angry texts, 13242 fear texts, and 12483 happy text.

Eliminate the Unnecessary Columns
Raw data contained columns that are not important in the process of determining emotional groups. Therefore, unnecessary columns were truncated using Python's 'pop' syntax.

Modelling
In order to discover the best model, Random Forest, Support Vector Machine (SVM), and Decision Tree were performed for the classification of emotion. The three models were performed using 63261 EEG vector features and were divided into 42384 and 20877 trained and tested data respectively.

Random Forest Model
The emotion classification of training data and test data used Random Forest Model from the matrix above, after a machine learning process using random forest, we get: 5698 sadness texts, 3957 angry texts, 3783 fear texts, and 6403 happy texts that match. After that, we analysed the predictive of accuracy data train and data test using the random forest algorithm. The following is table and bar chart of the accuracy of data train and data test predictive that had comparative with the real manually labelling.

Manual Train Classify Predictive Train Classify
Manual Test Classify Predictive Test Classify

Support Vector Machine (SVM)
In the same steps as we did with the random forest method, we took a 63261 tweets dataset in SVM, and two participants did the process labeling manually. Furthermore, we got the 19990 sadness texts, the 17546 angry texts, the 13242 fear texts, and the 12483 happy texts. Using the SVM model, we analyzed the accuracy of the predictive emotional classification versus the manually labeled baseline data. The following is the table and bar chart of accuracy results.  The results obtained are not good, because the accuracy of training data and test data only reaches around 32%.

Decision Tree
The basic ideas of the decision tree algorithm are as follows: (1) Choose the best attribute by using Attribution Selection Measures (ASM) to split notes; (2) Make the attribute a decision node and separate the dataset into smaller subsets; (3) Start building the tree by repeating this process recursively for each child until one of the conditions will match of all tuples have the same attribute value, there are no more attributes left, and here are no more examples. Below the result of the accuracy of the predictive emotional classification versus the manually label data.  The level of accuracy of training data with decision tree techniques is 100% for training data and 87% for test data. Besides, the Confusion Matrix reference was carried out to evaluate the performance, which also represents the prediction and actual conditions of the data generated by the ML algorithm. In addition, the Matrix determines the Accuracy, Precision, Recall and Specificity, as a result, it needs to know the most representative model of the writers and readers' emotions with the same text. Annotation was conducted manually and the reaction of the reader's brain waves was used to detect the accuracy level.  The table 5 showed the accuracy results in classifying the emotions generated by the three models as well as in the training and test data. In addition, the confusion matrix used to see how well or how much accuracy generated from the classification model has made to predict or classify classes from testing data, as shown in table 6. There were modeling changes in the brain waves for the four bases of emotion, in which the random forest technique has the best results, compared to the SVM and decision trees. In SVM model, the evaluation results obtained are very small, because there are weaknesses present when applied to large data which will be controlled when enriched by adding kernel techniques and greedy-search. However, when compared to the decision tree model, the results are not too large, because the two techniques are random forest, which have the same algorithm and serve as an extended decision tree. Another insight is that, the angry statement has the highest accuracy value among the four emotions, because it is easier to read in anger, therefore, the brain waves also become more legible for emotional expressions.
In this research, the accuracy value of the SVM was low, however in some papers it shows a good significant value both for facial expression stimuli (Huang et al., 2017), and for detecting emotional words in a linear and direct way (Bo Pang and Lillian Lee 2019).
Theoretically, this research contributes to the field of psychology in terms of recognizing someone's writing, and the effect of affecting the reader's emotions. Meanwhile, practically, examining text with brain waves has its own difficulties, because the results are not satisfactory when the reader does not understand deeply the writing. Therefore, it will be easier to detect emotions with the help of a voice or video expressing the writing.

Conclusion
Analyzing of emotion recognition from brain signals using 14 EEG channels was proposed and the result showed that the emotion recognition with text, based on stimuli from brain EEG channels is possible. This showed a high accuracy of random forest over decision tree and SVM in all selected subjects. Also, despite the absence of strong emotion due to physiological indication to correlate the brain activity at the cortical level, it wasdiscovered that brain wave approach indicates the possibility of recognition. However, this research mainly focused on the analyzing model and classification techniques that could be used for EEG signal processing. Therefore, future work should look at using a fusion approach for recognizing emotion.