Topic-Dependent Sentiment Analysis of
Financial BlogsNeil O’Hare, Michael Davy, Adam Beringham, Paul Fergusion,
Paraic Sheridan, Cathal Gurrin, Alan F. SmeatonDate: 2010/04/19
Speaker: Yu-Cheng Hsieh
Outline
• Introduction• Glossary• Issues• Development of corpus• Analysis of corpus• Topic-based analysis• Experiment & Result• Conclusion
Introduction
• No existing work used blogs as source, most work used news as source.
• News are more likely to report a stock’s past performance.
• Blogs are more likely to express opinions and to make predictions about the performance of stocks.
Introduction (Cont.)
• The aim is to…1. Automatically extract the
subjective opinions uniquely found on blogs.
2. Track the changing sentiment from the blogosphere towards individual stocks and the market in general.
• Supervised
Glossary
• Document: a blog article.
• Topic: name of a stock.
• Unique document: a document contains a topic only.
• Topic shift: an issue in a multiple topic document.
Glossary (Cont.)
• Doc-Topic pair: a topic in a non-unique document. (also a sub-document of a document)
• Inter-annotator agreement: the agreement level of annotating labels on an object.
Issues
• Topic Shift - How to extract those topics in the document?
• What level should be analyzed? Document level? Paragraph level? sentence level? word level?
• How many labels should be used to annotate?
Extract sub-document
• Using proximity approach- Steps1. Find out topic word: T2. Set a window size: N3. Starting from T, expanding N words both
at the right and left side of T.
Development of corpus
• The corpus is made up of financial blog articles from “blogged.com”
• 232 financial blogs are identified
• Separate articles in blogs into 2 crawls according to the date
- Craw1: 3 weeks in Feb. 2009 - Craw2: 5 weeks from May. to Jun. 2009
Development of corpus (Cont.)
• Noise Removal - Using DiffPost algorithm - Concept: noise tend to be repeated across multiple articles.- Steps1. Brake each article into HTML segments2. Compare those segments3. Remove the repeat segments, only unique segment
s are kept.
Development of corpus (Cont.)
• Labels - Very Negative/Positive - Neutral - Negative/Positive - Mixed - Not relevant - IDK (I Don’t Know)
Development of corpus (Cont.)
• Topics and retrieval- 500 stocks were chosen to be topics from
“S&P 500”.
- Relevant articles must contain the whole company name in upper case.
- Unique annotations are identified by the combination of document and topic, doc-topic pair.
Development of corpus (Cont.)
• Topics and retrieval- Also annotate a number of documents with
respect to their sentiment towards stocks in general.
=> ~ 1526 unique doc-topic pairs. ~ 167 of which were annotated for stocks in general. ~ 164 of which were annotated by two
annotators to facilitate inter-annotator agreement
analysis.
Analysis of corpus
• Annotation statistics
Analysis of corpus(Cont.)• Inter-Annotator Agreement
Cohen’s Kappa
• Example- Probability of consistent agreement P(a)= (20+15)/50=0.7- A said YES 30 times => 30/50=0.6 B said YES 25 times => 25/50=0.5 probability for both said… YES = 0.6*0.5 =0.3, NO=0.4*0.5=0.2 =>Probability of random agreement P(e)=0.3+0.2=0.5- Kappa = (0.7-0.5)/(1-0.5)=0.4
Analysis of corpus(Cont.)• Topic Relevance
Topic-based sentiment analysis
• Topic-based text extraction
- Blog articles often contains multiple topics.
- Topic-based extraction enables sentiment analysis at sub-document level, this should alleviate the topic-shift problem.
Topic-based sentiment analysis(Cont.)
• Topic-based text extraction- Three approaches to extract sub-
document1. N-word extraction2. N-sentence extraction3. N-paragraph extraction
Topic-based sentiment analysis(Cont.)
• Sentiment classification- The classification task attempts to model a
function 1. For binary classification
2. For 3-point classification
pairtopicdocaisXYXf ,:
},{ negativepositiveY
},,{ neutralnegativepositiveY
Experiment
• Discarded those Doc-Topic in the corpus not having labels , or were labelled in inconsistently by more than one annotators.
• 687 labelled documents for binary classification• 917 labelled documents for 3-point classification• Compare three classifiers 1. Multinomial Naïve Baye 2.SVM
3. Trivial classifier as baseline• 10-fold validation• Performance metric: classification accuracy• Sub-document were used to train the classifier
Y
Results
• Document level only
Results (Cont.)
Results (Cont.)
Results (Cont.)
• Binary classification using MNB at N=30
Conclusion
• Explored the use of blog sources for sentiment analysis in the financial domain
• Developed a corpus of over 1,500 document-level annotations
• Analysis of the annotation effort suggets that humans have particular difficulty annotating for degree of polarity
• Proposed text-extraction approach to solve topic-shift problem.
• Plan to explore the use of linguistic features and domain independent experiments
Thanks for your listening