Slide 1

Subjectivity and Sentiment Analysis of Arabic Tweets with Limited Resources Supervisor Dr. Verena Rieser Presented By ESHRAG REFAEE OSACT 27 May 2014 Slide 2 Outline 1. Introduction The concept of subjectivity and sentiment analysis (SSA) Motivations and challenges of SSA for Arabic Previous work on SSA of Arabic social networks 2. Experimental setup Twitter corpus: collection and annotation Evaluation metrics Machine learners 3. Results and Error Analysis 4. Summary and future work 2 Slide 3 Subjectivity and Sentiment analysis (SSA) Definition: Analysing and understanding peoples sentiments, evaluations, opinions, attitudes, and emotions from written text. 3 Slide 4 Hierarchical Model of Subjectivity and Sentiment analysis (SSA) 4 User- generated text SubjectivePositiveNegativeObjective Slide 5 Applications In addition to its significance as a major sub-field of Natural Language Processing (NLP) research, SSA has a range of real-world applications: Commercial applications measuring success of a product Social applications Political applications Economical applications 5 Slide 6 SSA and Social Networks The growing importance of sentiment analysis coincides with the growth of social media such as micro-blogs. 6 Slide 7 7 Slide 8 Twitter (Statistic Brain, 2014) March 2012, Twitter now available in Arabic (Twitter Blog, 2012) 8 Twitter ~60 M tweets/day >600 M active users 10 th most popular site in the world SSA and Twitter Slide 9 About Arabic Arabic is the language of over 422 million people First language of the 22 member countries of the Arabic League Official language in three other countries (UNISCO, 2013). 9 Slide 10 About Arabic Arabic is the language of over 422 million people Arabic language can be classified into three major levels (Habash, 2010): Classic Arabic (CA) Modern standard Arabic (MSA) Arabic Dialects (AD). 10 Used in social networks side-by-side Slide 11 Challenges with Respect to Arabic Limited availability of NLP resources for DA. Noisy features. No large-scale Arabic Twitter corpus annotated for SSA publically available. Sparse labelled data. BUT: Lots of unlabelled data! 11 Slide 12 Challenges With Respect to Twitter Bad language (Eisenstein, J. 2013) Unclear sentiment indicator Dynamic nature/ topic-shifting (Go et al, 2009). 12 Equality in supressing personal freedom is justice ew, ugh instead of disgusting bro instead of brother Slide 13 Previous Work on SSA of Arabic Tweets PublicationFeature-setsClassificatio n scheme Results Abdul_magged et al (2012) Stem and lemma word tokens, POS, semantic features, user: person/org SVM (two-stage binary classification) The best acc. 65.32% for sentiment analysis and 79.01% for subjectivity analysis Mourad and Darwish (2013) Stem word tokens, tweets-specific features, stylistic features SVM and NB with 10-fold cross-validation The best acc. 64.1% for subjectivity classification and 72.5% for sentiment classification 13 Mainly Supervised Learning on manually annotated corpora. Costly annotations. Not scalable/ applicable to unseen topics! Slide 14 Previous Work on SSA of Arabic Tweets PublicationFeature-setsClassificati on scheme DatasetsResults Abdul_Mageed et al (2012) Stem and lemma word tokens, POS, semantic features, user: person/org SVM (two- stage binary classification) 3k Arabic tweets The best acc. 65.32% for sentiment analysis and 79.01% for subjectivity analysis Mourad and Darwish (2013) Stem word tokens, tweets- specific features, stylistic features SVM and NB with 10-fold cross- validation 2,300 Arabic tweets The best acc. 64.1% for subjectivity classification and 72.5% for sentiment classification 14 Word-based features. SVM shown to perform best (large feature sets) Evaluation: 10-fold cross-validation Held-out test set from same corpus No test for unseen topics/ scalability for topic shift! Slide 15 Outline 1. Introduction Motivations and challenges of subjectivity and sentiment analysis (SSA) for Arabic Previous work on SSA of Arabic social networks 2. Experimental setup Twitter corpus: collection and annotation Evaluation metrics Machine learners 3. Results and Error Analysis 4. Summary and future work 15 Slide 16 Methodology and Approach Un- labelled tweets Human annotators Gold- standard labelled tweets Arabic ALP tools Train machine learning scheme: SVM classifier Manually- annotated held-out test set Features Model evaluation Slide 17 Arabic Twitter SSA Corpora 17 Slide 18 Arabic Twitter SSA Corpora: Gold Standard Data Set Manually annotated for sentiment analysis (total=3,309) 2 native speaker annotators (weighted Kappa=0.76) 18 Slide 19 Arabic Twitter SSA Corpora: Held-out Test Set 963 tweets were manually annotated for evaluating the trained models. 19 Slide 20 Arabic Twitter SSA Corpora Sentiment labelExample Positive Tourism in Yemen, unbelievable beauty Negative Unfortunately, we use the iPhone Neutral Merkel calls for Ukraine to form a new government 20 Examples of annotated tweets Slide 21 Features Extraction TypeLinguistic tool/resourceFeature-set Morphological features Arabic morphological analyser: MADA + TOKAN V3.2 (Habash and Rambow, 2005 & Habash, and Roth, 2009). Diacritic, Aspect, Gender, mood, person,part-of- speech, State, voice, Has- morph-analysis Syntactic features N-grams of word tokens Semantic features Polarity lexicons: 1)ArabSenti (Abdul- Mageed et al, 2011) 2)MPQA-translation (Wilson et al, 2005) Has-positive-lexicon, Has-negative-lexicon, Has-neutral-lexicon, Has-negator Stylistic features Has-positive-emoticon, Has-negative-emoticon 21 Slide 22 Subjectivity and Sentiment Classification Experiments 22 Slide 23 SSA Classification: Problem Formulations 23 TextSubjectivePositiveNegativeObjective TextPositiveNegativeNeutral Slide 24 Machine Learning Classifiers Support Vector Machines (SVM): Sequential Minimal Optimization-SMO (Platt, 1999) Majority baseline: ZeroR 24 SVM aims to identify the Optimal hyperplane that linearly separates data instances with the maximum margin (Hsu et al, 2003) Slide 25 Evaluation Metrics F-measure Accuracy: Significant differences: T-test with p