We connect, inspire and guide women in computing and organizations that view technology innovation as a strategic imperative.

Predicting User Demographics, Emotions and

Opinions in Social Networks Svitlana Volkova

Center for Language and Speech Processing, Johns Hopkins University

(now at Pacific Northwest National Laboratory) svitlana.volkova@pnnl.gov

ABSTRACT Social networks are virtual environments where people express their thoughts, emotions, and opinions. We analyze a large sample of 123,000 Twitter users and 25 million tweets to investigate the relation between user emotions and predicted demographics. Our methodology is based on building machine learning models to predict emotions and demographic profiles from user content. We report novel demographic-affect correlations and its implications on online self-disclosure.

AUDIENCE [Natural Language Processing] [Machine Learning] [Social Network Analysis] [Opinion Mining] [Emotion Detection] [Demographic Classification] [Advanced Technical Talk]

INTRODUCTION Twitter and Facebook are prominent social networks, used regularly by over 1/7th of the world's population. Researchers used the massive volumes of data to study how users present themselves and the language they use [1], showing how to predict user psycho-demographic profiles [2,3], user emotions [4], and well-being. This study analyzes user communications in a social network on a large scale—25 million tweets, 123,513 user profiles—examining a range of automatically detected emotions, opinions, and a variety of demographic traits. This work can help social network users to understand how others may perceive them based on how they communicate in social media, in addition to its evident applications in online sales and marketing, targeted advertising, large-scale polling, and healthcare analytics.

DATA Focusing on Twitter, we use crowdsourcing to get demographic labels for a sample of U = 5,000 users (Table 1), and train machine learning classifiers to predict

these demographic traits from the textual content generated by these users. We then apply attribute classifiers to get labels for a much larger sample of U = 123,513 users. We use a similar method for labeling emotions expressed in user text. We train an emotion classifier on an initial sample of TE = 52,925 tweets, then use the classifier to get emotion labels for a much larger sample of T = 24,919,528 tweets. However, rather than obtaining emotion tags for the initial sample through crowdsourcing, we use tweets annotated with emotional hashtags such as #disgust or #anger, identifying a specific emotion. To perform a reliable analysis of the differences in emotion expressed by different user groups, our demographic and emotion predictions must be highly accurate. We report that our models for emotion and demographic classification outperform the existing state-of-the-art systems. Our high-level methodology is shown in Figure 1.

Figure 1. Our approach for predicting user demographics, emotions, and opinions in social media.

Attribute Binary Attribute Values Gender Male: 2,124, Female: 2,874 Age Below 25: 2,511, Above 25: 1,372 Ethnicity Afr. Amer.: 1,705, Caucasian: 2,409 Education High school: 3,423, Degree: 1,575 Income <$35K: 3,324, >$35K: 1,675 Children Yes: 797, No: 4203 Optimism Pessimist: 907, Optimist: 2,655 Life Satisfaction Dissatisfied: 840, Satisfied: 2,949

Table 1. Profile annotations collected via crowdsourcing.

We connect, inspire and guide women in computing and organizations that view technology innovation as a strategic imperative.

METHOD Building Classifiers for Affect Prediction We define two tweet-based supervised models Φ! t and Φ! t for emotion and sentiment classification, trained on tweets labeled with affect classes. We represent each tweet t! as a feature vector over the words used in a tweet and add other stylistic and syntactic features [2,4]. Then, we train a log-linear model [5] (or logistic regression) from a labeled feature vector to map each unlabeled tweet to the most likely variable assignment using a parameter vector θ! as shown below for the emotion classifier:

Φ! t = argmax!P E t = e t, θ! (1)

P E t = e t, θ! = 1 + exp  (−θ! ∙ f t!!

.

Inferring User Demographics in Social Media Earlier work on predicting user attributes on Twitter has used supervised models with lexical bag-of-word features for classifying four demographic attributes: gender, age, political preferences, and ethnicity [1,6]. We focus on previously underexplored personal attributes including education level, annual income, optimism, and life satisfaction. We also investigate the relation between emotions, opinions, and demographic attributes on Twitter.

Building Classifiers for Attribute Prediction We define supervised models Φ! u for classifying user attributes presented in Table 1. These log-linear models [5] are learned from 𝑇 = 200 tweets per user. We construct a feature vector representation f 𝑇 from a word distribution over 𝑇 tweets per user 𝑢. The models then map each unlabeled user represented as f 𝑇 to the most likely variable assignment as defined below:

Φ! u = argmax!P A u = a 𝑇, θ! (2)

P A u = a 𝑇, θ! = 1 + exp  (−θ! ∙ f 𝑇!!

.

Measuring User Emotions and Sentiments Given a set of tweets with predicted emotions and sentiments, we estimate the proportion or normalized frequency of each emotion 𝑒 and sentiment 𝑠 per user. We then calculate positive emotion and sentiment scores: 𝐸! 𝑢 = 𝑒!"# − 𝑒!"#$% − 𝑒!"# − 𝑒!"#$ − 𝑒!"#$ (3) 𝑆! 𝑢 = 𝑠!"# − 𝑠!"#. (4)

Estimating Emotion and Opinion Differences We first evaluate whether Twitter users of contrasting attributes (e.g., a! = Male vs. a! = Female) differ in the emotions and sentiments they tend to express. We examine the distributions of the emotions e ∈ E and the sentiments s ∈ S for contrasting populations. We apply a non-parametric Mann-Whitney U test. Our null hypothesis H! is that the mean emotion and sentiment scores for the population a! are the same as for a!. The alternative hypothesis H! is that these two user populations have different mean scores.

We then quantitatively measure emotion and opinion differences between contrastive populations. For that we estimate the average scores for emotions and opinions for the populations: µμ!

!! = 𝑒!!! /𝑈 and µμ!!! = 𝑒!!! /

𝑈, e! ∈ E (similarly, µμ!!! and µμ!

!! , s! ∈ S). We take the difference between mean values estimated over groups of a! and a! users for every emotion Δµμ! = µμ!

!! − µμ!!! and

opinion Δµμ! = µμ!!! − µμ!

!! .

RESULTS Figure 2 presents classification results in terms of the area under the receiver operating characteristic curve (ROC AUC) for the attributes outlined in Table 1. Our results for gender and ethnicity prediction demonstrate significantly higher performance compared to previous work [1,3,6].

Figure 2. Attribute prediction results in ROC AUC.

We estimate sentiment prediction quality on 3,223 tweets released as an official SemEval-2013 test set. We evaluate emotion prediction quality using 10-fold cross validation (Table 2). Performance is reported in terms of the weighted F-score – F1 = 0.6 for sentiment (3 classes) and F1 = 0.78 for emotion (6 classes).

Emotions Mohammad [7] Wang [8] This Work Anger 0.28 0.72 0.80 Disgust 0.19 – 0.92 Fear 0.51 0.44 0.77 Joy 0.62 0.72 0.79 Sadness 0.39 0.65 0.62 Surprise 0.45 0.14 0.64 All 0.49 – 0.78

Table 2. Emotion prediction results in F1 score compared to other systems [7,8].

ANALYSIS We now analyze emotional differences between users with contrasting attributes and present the results in Figure 3. Our key findings are discussed in detail. All the differences are stat. sign. different (p-value < 0.001). Gender: Female users generate more happy and sad tweets, while male users produce more surprise and fear tweets. In line with previous work, our results further confirms that female users are more emotional online.

AgeChildrenOptimism

Life SatisfactionIncome

EducationGenderEthnicity

ROC AUC

0.0

0.2

0.4

0.6

0.8

1.0

0.63

0.72

0.72

0.72

0.73

0.77

0.9

0.93

We connect, inspire and guide women in computing and organizations that view technology innovation as a strategic imperative.

Age: Older (over 25 years old) users are 7.5% more positive and generate 4% more joyful tweets and 4% fewer sad tweets. Younger users produce more disgust, anger and surprise tweets. Our results are in line with the recently explored “aging positivity effect” in social media that states that older people are happier than younger people. Education: Users with a degree produce 4.7% more joyful tweets. In contrast, users with only high school education generate 4.4% more sad, disgust and angry tweets.

a. Gender b. Age c. Education d. Kids

e. Income f. Life Satis. g. Optimism

Figure 3. Emotional differences across demographics.

Inferring User Demographics from Affects Figure 4 shows the predictive power (reported as regression coefficients) of emotion and sentiment features for inferring user demographics.

Figure 4. Predicting demographics from affects.

We show that some affects are predictive of one attribute value (red: male, satisfied with life, optimist, no kids, older, higher income, degree), some of an opposite value (blue). For instance, negative sentiment and sadness are predictive of users with no children and older users. We also draw a dendrogram for attributes (columns) and emotions (rows) to visualize cross-attribute and cross-affect similarities.

CONCLUSION In this work we demonstrate how social media can be effectively used to analyze user emotions and sentiments,

and to infer latent online identities. We also show that users of different demographics project different emotions on Twitter. Some of our results are stereotypical: e.g., users with high income produce significantly less sad tweets and users with lower income express more negative emotions;

female users are more emotional and opinionated than male users; older users express more joy and less sadness than younger users. We hope that this work would allow users to see how they may be perceived by their peers in social networks, so they can better understand what drives the image they project online.

PARTICIPATION STATEMENT I will attend the conference if accepted.

BIO Svitlana Volkova is a Research Scientist in the Data Sciences and Analytics Group, Pacific Northwest National Laboratory. She received her PhD in Computer Science from Johns Hopkins University. She was affiliated with the Center for Language and Speech Processing. Her PhD research focused on developing machine learning models for streaming social media analytics, fine-grained emotion detection, and multilingual sentiment analysis. She interned at Microsoft Research in 2011, 2012 and 2014. She was awarded the Google Anita Borg Memorial Scholarship in 2010 and the Fulbright Scholarship in 2008.

REFERENCES [1] Schwartz A, Eichstaedt C. Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one, 8 (9), e73791. 2013. [2] Volkova S, Bachrach Y, Armstrong M, Sharma V. Inferring latent user properties from texts published in social media. In Proceedings of AAAI. 2015. [3] Kosinski M, Stillwell D, Graepel T. Private traits and attributes are predictable from digital records of human behavior. National Academy of Sciences 2013. [4] Volkova S, Bachrach Y. On Predicting Sociodemographic Traits and Emotions from Communications in Social Networks and Their Implications to Online Self-Disclosure. Cyberpsychology, Behavior, and Social Networking 18.12 (2015. [5] Smith NA. Tutorial: Log-linear models (2004). [6] Zamal A, Liu W, & Ruths D. Homophily and latent attribute inference: Inferring latent attributes of Twitter users from neighbors. In Proceedings of ICWSM. 2012. [7] Mohammad S, Kiritchenko S. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence 2014. [8] Wang W, Chen L, Thirunarayan K, Sheth P. Harnessing Twitter “big data” for automatic emotion identification. In Proceedings of SocialCom. 2012.

joy

sadnessdisgustanger

surprisefearscore

Emotion Difference, %

-4 -2 0 2 4

-0.1

-2.7

1.1

2.6

-1

Male

Female

joy

sadnessdisgustanger

surprisefearscore

Emotion Difference, %

-10 -5 0 5 10

-4

4.3

0.5

0.4

0.5

-1.7

-7.5

Above 25

Below 25joysadnessdisgustanger

surprisefearscore

Emotion Difference, %

-10 -5 0 5 10

4.7

-4.4

-0.8

-0.7

-0.7

1.9

8.7School

Degree

joy

sadnessdisgustanger

surprisefearscore

Emotion Difference, %

-5 0 5 10

3.6

-3.5

-0.2

-0.3

-0.9

1.2

6.4

No Kids

Kids

joy

sadnessdisgustanger

surprisefearscore

Emotion Difference, %

-10 -5 0 5

-4.9

4.5

0.7

0.6

0.8

-1.8

-8.9

>$35K

<$35Kjoysadnessdisgustanger

surprisefearscore

Emotion Difference, %

-20

-15

-10 -5 0 5 10 15

5.7

-2.9

-1

-1.1

-0.4

-0.4

11.1

Dissatisfied

Satisfied

joy

sadnessdisgustanger

surprisefearscore

Emotion Difference, %

-15

-10 -5 0 5 10 15 20

6.9

-3.4

-1

-1.3

-0.5

-0.7

13.3

Pessimist

Optimist

Income

Education

Ethnicity

Optimism

Life.Statisf

Gender

Age

Children

NegativeSadnessDisgustAngerSurpriseFearNeutralSentScorePositiveEmoScoreJoy

-2 -1 0 1 2Column Z-Score

08

Color Keyand Histogram

Count