sentiment classification with case-based reasoning

23
Case-Based Approach to Cross- Domain Sentiment Classification ICCBR - Sep/2012 Bruno Ohana Sarah-Jane Delany Brendan Tierney Dublin Institute of Technology - Ireland

Upload: bohanairl

Post on 05-Dec-2014

3.228 views

Category:

Documents


3 download

DESCRIPTION

Given at ICCBR conference Sep/2012, Lyon, France.

TRANSCRIPT

Page 1: Sentiment Classification with Case-Based Reasoning

Case-Based Approach to Cross-Domain Sentiment Classification

ICCBR - Sep/2012

Bruno OhanaSarah-Jane Delany

Brendan Tierney

Dublin Institute of Technology - Ireland

Page 2: Sentiment Classification with Case-Based Reasoning

Outline

● Sentiment Classification

● Domain Dependence

● Lexicon-based methods.

● Case Based Approach

● Experiment and Results.

Page 3: Sentiment Classification with Case-Based Reasoning

Sentiment Classification● For a given piece of text, determine sentiment

orientation.

● Positive or Negative?

“This is by far the worst hotel experience i've ever had. the owner overbooked while i was staying there (even though i booked the room two months in advance) and made me move to another room, but that room wasn't even a hotel room!”

Page 4: Sentiment Classification with Case-Based Reasoning

Applications● Search and Recommendation Engines.

○ Show only positive/negative/neutral.

● Market Research.○ What is being said about brand X on Twitter?

● Ad Placement.

● Mediation of online communities.

Page 5: Sentiment Classification with Case-Based Reasoning

Domain Dependence

Supervised Learning Methods

● Good Performance, but:○ Labeled data is Expensive.○ Availability for all domains unlikely.

● Classifiers are domain specific.○ Ex: “Kubrick” may be a good opinion predictor for film

reviews, but not on other domains.

● (Aue & Gamon '05)○ Straightforward Train/Test across domains yields poor

results.

Page 6: Sentiment Classification with Case-Based Reasoning

Using a Sentiment LexiconDatabase of terms associated with positive or negative sentiment.

● Manual: General Enquirer (Stone et al '67)● Corpus Based (Hatzivassiloglou & McKeown '97)● Lexical Induction: SentiWordNet (Esuli et al '06)● Some sample sizes:

○ GI: 4K○ SWN: 26K

Approach:● Scan document for term ocurrences, prediction based

on agregated results for positive/negative classes.

● No need for Training data sets.

Page 7: Sentiment Classification with Case-Based Reasoning

Sentiment Classification with Lexicons

POS Tagger NegEx Classifier Prediction

Sent.Lexicon

Lexicon-Based classification

● Annotate text with POS and negation information.● Identify words present on lexicon.

○ Retrieve numerical score from lexicon indicating opinion.

● Aggregate results, use a rule to make prediction.○ Ex: max(PosScore,NegScore)

Page 8: Sentiment Classification with Case-Based Reasoning

Sentiment Classification with Lexicons

The/DT computer-animated/JJ comedy/NN ''/'' shrek/NN ''/'' is/VBZ designed/VBN to/TO be/VB enjoyed/VBN on/IN different/JJ levels/NNS by/IN different/JJ groups/NNS ./. for/IN children/NNS ,/, it/PRP offers/VBZ imaginative/JJ visuals/NNS ,/, appealing/VBG new/JJ characters/NNS mixed/VBN with/IN a/DT host/NN of/IN familiar/JJ faces/NNS ,/, loads/NNS of/IN action/NN and/CC a/DT barrage/NN of/IN big/JJ laughs/NNS

The computer-animated comedy "shrek" is designed to be enjoyed on different levels by different groups . for children , it offers imaginative visuals , appealing new characters mixed with a host of familiar faces , loads of action and a barrage of big laughs

Page 9: Sentiment Classification with Case-Based Reasoning

Lexicon-Based Classification: Issues● Performance of supervised learning methods is better.

● Selection of lexicon, classifier are established upfront.○ Ex: Use SWN with classifier F.○ Your choice can be sub-optimal.

● Lexicons perform differently on different domains. (Ohana et al, '11)

Page 10: Sentiment Classification with Case-Based Reasoning

Sentiment Classification with Lexicons

POS Tagger NegEx Classifier Prediction

Sent.Lexicon

Classifier Considerations

● Which Sentiment Lexicon to Use?● How to apply term sentiment information to the document?

○ What part-of-speech to use.○ Enable/Disable Negation Detection.○ How to count terms? (once, every time, adjust for

frequency)

ClassifierClassifier

Sent.LexiconSent.

Lexicon

Page 11: Sentiment Classification with Case-Based Reasoning

Our ApproachBuild a case-base using out-of-domain data where:

● Problem description maps to document characteristics.

● Solution description maps to successful combinations of lexicons/classifiers.

Use case base to decide on which lexicon and classifier to use on a new document/domain.

Page 12: Sentiment Classification with Case-Based Reasoning

Experiment - Case RepresentationProblem Description

Solution Description● Set of lexicons S={L1,...Ln} that yielded a correct prediction on input

document.● We use 5 different lexicons from the literature.

Counts for words, tokens and sentences; Avg. sentence size

Part-of-speech frequencies.

Counts for total Syllable and Monosyllable count.

Spacing ratio; Word-token ratio.

Stop words ratio.

Unique words count.

Page 13: Sentiment Classification with Case-Based Reasoning

Experiment - Data Sets

User generated reviews on 6 x domains● English, Plain text.● Balanced classes.● Borderline cases removed.

Data Set Size Source

Hotels 2874 Tripadvisor

Films 2000 IMDB

Electronics 2072 Amazon.com

Music 5902 Amazon.com

Books 2034 Amazon.com

Apparel 566 Amazon.com

Page 14: Sentiment Classification with Case-Based Reasoning

Experiment - Case Base

6 x domains.● Customer reviews in raw text.● Build 6 x case-bases of 5 x domains (Leave one out).

Movies

Electronics

Apparel

Hotels

Books

Music Albums

Page 15: Sentiment Classification with Case-Based Reasoning

Building the Case Base

Page 16: Sentiment Classification with Case-Based Reasoning

Experiment - Case Bases

Case creation:● Found at least one lexicon that gives a correct

prediction.Left out Domain Case Base Size % Positive % Negative

Books 9683 53.3 46.7

Electronics 9592 53.6 46.4

Film 9614 54.1 45.9

Music 6137 52.6 47.4

Hotels 11516 53.5 46.5

Apparel 11002 53.4 46.6

Page 17: Sentiment Classification with Case-Based Reasoning

Lexicons in Case Solution

Page 18: Sentiment Classification with Case-Based Reasoning

Experiment - Retrieval and Ranking

● K-NN and Euclidean Distance.

● Ranking: Select most common Lexicon out of K cases retrieved.

Solutions (k=3) Ranking (Count) Selected

case1 = {L1,L3,L4} L1 (3) {L1}

case2 = {L1,L2} L3 (2)

case3 = {L1,L3,L5} L2, L4, L5 (1)

Page 19: Sentiment Classification with Case-Based Reasoning

Case Based Approach

Page 20: Sentiment Classification with Case-Based Reasoning

Experiment Results

Baseline Results● Results for lexicon that performed best in domain (out of 5

lexicons)

Page 21: Sentiment Classification with Case-Based Reasoning

Summary

Case Based Approach● Selection of lexicon/classifier up to case-base.

● Expandable.○ Easy to add more lexicons, classifiers, cases.

● Experimental results beat best-lexicon baseline in 4 of 6 domains.

Page 22: Sentiment Classification with Case-Based Reasoning

Next Steps

Grow Solution Search Space● More lexicons, more classifiers.

Retrieval and Ranking● For larger search space, will not scale.● Room to improve case problem description.

Case Base Creation● Add negative results instead of discarding.

Page 23: Sentiment Classification with Case-Based Reasoning

Thank You.