opinion mining

Opinion Mining

Sudeshna Sarkar

24th and 26th October 2007

Janyce Wiebe, U. PittsburghClaire Cardie, Cornell U.

Ellen Riloff, U. UtahJosef Ruppenhofer, U. Pittsburgh

Bing Liu

MIAS - June 14, 2007 3

Introduction – facts and opinions

Two main types of information on the Web. Facts and Opinions

Current search engines search for facts (assume they are true) Facts can be expressed with topic keywords.

Search engines do not search for opinions Opinions are hard to express with a few keywords

How do people think of Motorola Cell phones?

Current search ranking strategy is not appropriate for opinion retrieval/search.

MIAS - June 14, 2007 4

Introduction – user generated content

Word-of-mouth on the Web One can express personal experiences and opinions on almost

anything, at review sites, forums, discussion groups, blogs ..., (called the user generated content.)

They contain valuable information Web/global scale

No longer limited to your circle of friends Our interest: to mine opinions expressed in the user-generated

content An intellectually very challenging problem. Practically very useful.

MIAS - June 14, 2007 5

Introduction – Applications

Businesses and organizations: product and service benchmarking. Market intelligence. Business spends a huge amount of money to find consumer

sentiments and opinions. Consultants, surveys and focused groups, etc

Individuals: interested in other’s opinions when Purchasing a product or using a service, Finding opinions on political topics, Many other decision making tasks.

Ads placements: Placing ads in user-generated content Place an ad when one praises an product. Place an ad from a competitor if one criticizes an product.

Opinion retrieval/search: providing general search for opinions.

MIAS - June 14, 2007 6

Question Answering

Opinion question answering:

Q: What is the international reaction to the reelection of Robert Mugabe as President of Zimbabwe?

A: African observers generally approved of his victory while Western Governments denounced it.

MIAS - June 14, 2007 7

More motivation

Product review mining: What features of the ThinkPad T43 do customers like and which do they dislike?

Review classification: Is a review positive or negative toward the movie? Tracking sentiments toward topics over time: Is anger ratcheting up or

cooling down? Etc.

MIAS - June 14, 2007 8

Two types of evaluation

Direct Opinions: sentiment expressions on some objects, e.g., products, events, topics, persons E.g., “the picture quality of this camera is great” Subjective

Comparisons: relations expressing similarities or differences of more than one object. Usually expressing an ordering. E.g., “car x is cheaper than car y.” Objective or subjective.

MIAS - June 14, 2007 9

Opinion search (Liu, Web Data Mining book, 2007)

Can you search for opinions as conveniently as general Web search?

Whenever you need to make a decision, you may want some opinions from others, Wouldn’t it be nice? you can find them on a search

system instantly, by issuing queries such as Opinions: “Motorola cell phones” Comparisons: “Motorola vs. Nokia”

Cannot be done yet!

MIAS - June 14, 2007 10

Typical opinion search queries Find the opinion of a person or organization (opinion

holder) on a particular object or a feature of an object. E.g., what is Bill Clinton’s opinion on abortion?

Find positive and/or negative opinions on a particular object (or some features of the object), e.g., customer opinions on a digital camera, public opinions on a political topic.

Find how opinions on an object change with time. How object A compares with Object B?

Gmail vs. Yahoo mail

MIAS - June 14, 2007 11

Find the opinion of a person on X

In some cases, the general search engine can handle it, i.e., using suitable keywords. Bill Clinton’s opinion on abortion

Reason: One person or organization usually has only one

opinion on a particular topic. The opinion is likely contained in a single document. Thus, a good keyword query may be sufficient.

MIAS - June 14, 2007 12

Find opinions on an object X

We use product reviews as an example: Searching for opinions in product reviews is different

from general Web search. E.g., search for opinions on “Motorola RAZR V3”

General Web search for a fact: rank pages according to some authority and relevance scores. The user views the first page (if the search is perfect). One fact = Multiple facts

Opinion search: rank is desirable, however reading only the review ranked at the top is dangerous

because it is only the opinion of one person. One opinion Multiple opinions

MIAS - June 14, 2007 13

Search opinions (contd)

Ranking: produce two rankings

Positive opinions and negative opinions Some kind of summary of both, e.g., # of each

Or, one ranking but The top (say 30) reviews should reflect the natural distribution

of all reviews (assume that there is no spam), i.e., with the right balance of positive and negative reviews.

Questions: Should the user reads all the top reviews? OR Should the system prepare a summary of the reviews?

MIAS - June 14, 2007 14

Reviews are similar to surveys

Reviews can be regarded as traditional surveys. In traditional survey, returned survey forms are treated

as raw data. Analysis is performed to summarize the survey

results. E.g., % against or for a particular issue, etc.

In opinion search, Can a summary be produced? What should the summary be?

MIAS - June 14, 2007 15

Roadmap

Opinion mining – the abstraction Domain level sentiment classification Sentence level sentiment analysis Feature-based sentiment analysis and

summarization Comparative sentence and relation extraction Summary

MIAS - June 14, 2007 16

Opinion mining – the abstraction(Hu and Liu, KDD-04)

Basic components of an opinion Opinion holder: A person or an organization that holds an specific

opinion on a particular object. Object: on which an opinion is expressed Opinion: a view, attitude, or appraisal on an object from an opinion

holder. Objectives of opinion mining: many ... We use consumer reviews of products to develop the

ideas. Other opinionated contexts are similar.

MIAS - June 14, 2007 17

Object/entity Definition (object): An object O is an entity which can be

a product, person, event, organization, or topic. O is represented as a tree or taxonomy of components (or parts), sub-components, and so on. Each node represents a component and is associated with a set of

attributes. O is the root node (which also has a set of attributes)

An opinion can be expressed on any node or attribute of the node.

To simplify our discussion, we use “features” to represent both components and attributes. The term “feature” should be understood in a broad sense

Product feature, topic or sub-topic, event or sub-event, etc Note: the object O itself is also a feature.

MIAS - June 14, 2007 18

A model of a review

An object is represented with a finite set of features, F = {f1, f2, …, fn}. Each feature fi in F can be expressed with a finite set of words or

phrases Wi, which are synonyms.

That is to say: we have a set of corresponding synonym sets W = {W1,

W2, …, Wn} for the features.

Model of a review: An opinion holder j comments on a subset of the features Sj F of an object O.

For each feature fk Sj that j comments on, he/she

chooses a word or phrase from Wk to describe the feature,

expresses a positive, negative or neutral opinion on fk.

MIAS - June 14, 2007 19

Opinion mining tasks

At the document (or review) level:

Task: sentiment classification of reviews Classes: positive, negative, and neutral Assumption: each document (or review) focuses on a single object O

(not true in many discussion posts) and contains opinion from a single opinion holder.

At the sentence level:

Task 1: identifying subjective/opinionated sentences Classes: objective and subjective (opinionated)

Task 2: sentiment classification of sentences Classes: positive, negative and neutral. Assumption: a sentence contains only one opinion

not true in many cases. Then we can also consider clauses.

MIAS - June 14, 2007 20

Opinion mining tasks (contd)

At the feature level:

Task 1: Identifying and extracting object features that have been commented on in each review.

Task 2: Determining whether the opinions on the features are positive, negative or neutral in the review.

Task 3: Grouping feature synonyms.

Produce a feature-based opinion summary of multiple reviews (more on this later).

Opinion holders: identify holders is also useful, e.g., in news articles, etc, but they are usually known in user generated content, i.e., the authors of the posts.

MIAS - June 14, 2007 21

More at the feature level

Problem 1: Both F and W are unknown. We need to perform all three tasks:

Problem 2: F is known but W is unknown. All three tasks are needed. Task 3 is easier. It

becomes the problem of matching discovered features with the set of given features F.

Problem 3: W is known (F is known too). Only task 2 is needed.

F: the set of featuresW: synonyms of each feature

MIAS - June 14, 2007 22

Roadmap

Opinion mining – the abstraction Document level sentiment classification Sentence level sentiment analysis Feature-based sentiment analysis and summarization Comparative Sentence and relation extraction Summary

MIAS - June 14, 2007 23

Sentiment classification

Classify documents (e.g., reviews) based on the overall sentiments expressed by authors, Positive, negative, and (possibly) neutral Since in our model an object O itself is also a feature, then

sentiment classification essentially determines the opinion expressed on O in each document (e.g., review).

Similar but different from topic-based text classification. In topic-based text classification, topic words are important. In sentiment classification, sentiment words are more important,

e.g., great, excellent, horrible, bad, worst, etc.

MIAS - June 14, 2007 24

Unsupervised review classification(Turney, ACL-02)

Data: reviews from epinions.com on automobiles, banks, movies, and travel destinations.

The approach: Three steps Step 1:

Part-of-speech tagging Extracting two consecutive words (two-word phrases)

from reviews if their tags conform to some given patterns, e.g., (1) JJ, (2) NN.

MIAS - June 14, 2007 25

Step 2: Estimate the semantic orientation of the extracted phrases Use Pointwise mutual information

Semantic orientation (SO): SO(phrase) = PMI(phrase, “excellent”)

PMI(phrase, “poor”)

Using AltaVista near operator to do search to find the number of hits to compute PMI and SO.

)()(

)(log),(

21

21221 wordPwordP

wordwordPwordwordPMI

MIAS - June 14, 2007 26

Step 3: Compute the average SO of all phrases classify the review as recommended if average SO is

positive, not recommended otherwise.

Final classification accuracy: automobiles - 84% banks - 80% movies - 65.83 travel destinations - 70.53%

MIAS - June 14, 2007 27

Sentiment classification using machine learning methods (Pang et al, EMNLP-02)

The paper applied several machine learning techniques to classify movie reviews into positive and negative.

Three classification techniques were tried: Naïve Bayes Maximum entropy Support vector machine

Pre-processing settings: negation tag, unigram (single words), bigram, POS tag, position.

SVM: the best accuracy 83% (unigram)

MIAS - June 14, 2007 28

MIAS - June 14, 2007 29

Roadmap

Opinion mining – the abstraction Document level sentiment classification Sentence level sentiment analysis Feature-based sentiment analysis and

summarization Comparative sentence and relation

extraction Summary

MIAS - June 14, 2007 30

Sentence-level sentiment analysis

Document-level sentiment classification is too coarse for most applications.

Let us move to the sentence level. Much of the work on sentence level sentiment analysis focus on

identifying subjective sentences in news articles. Classification: objective and subjective. All techniques use some forms of machine learning. E.g., using a naïve Bayesian classifier with a set of data

features/attributes extracted from training sentences (Wiebe et al. ACL-99).

MIAS - June 14, 2007 31

Using learnt patterns (Rilloff and Wiebe, EMNLP-03)

A bootstrapping approach. A high precision classifier is used to automatically identify some

subjective and objective sentences. Two high precision (low recall) classifiers were used,

a high precision subjective classifier A high precision objective classifier Based on manually collected lexical items, single words and n-

grams, which are good subjective clues. A set of patterns are then learned from these identified subjective

and objective sentences. Syntactic templates are provided to restrict the kinds of patterns to be

discovered, e.g., <subj> passive-verb. The learned patterns are then used to extract more subject and

objective sentences (the process can be repeated).

MIAS - June 14, 2007 32

Subjectivity and polarity (orientation) (Yu and Hazivassiloglou, EMNLP-03)

For subjective or opinion sentence identification, three methods were tried: Sentence similarity. Naïve Bayesian classification. Multiple naïve Bayesian (NB) classifiers.

For opinion orientation (positive, negative or neutral) (also called polarity) classification, it uses a similar method to (Turney, ACL-02), but with more seed words (rather than two) and based on

log-likelihood ratio (LLR). For classification of each word, it takes average of LLR

scores of words in the sentence and use cutoffs to decide positive, negative or neutral.

MIAS - June 14, 2007 33

Other related work

Consider gradable adjectives (Hatzivassiloglou and Wiebe, Coling-00)

Semi-supervised learning with the initial training set identified by some strong patterns and then applying NB or self-training (Wiebe and Riloff, CicLing 05)

Finding strength of opinions at the clause level (Wilson etal, AAAI-04)

Sum up orientations of opinion words in a sentence (or within some word window) Kim and Hovy, Coling-04

MIAS - June 14, 2007 34

Let us go further?

Sentiment classifications at both document and sentence (or clause) level are useful, but They do not find what the opinion holder liked and disliked.

An negative sentiment on an object does not mean that the opinion holder dislikes everything about

the object. A positive sentiment on an object

does not mean that the opinion holder likes everything about the object.

We need to go to the feature level.

MIAS - June 14, 2007 35

But before we go further

Let us discuss Opinion Words or Phrases (also called polar words, opinion bearing words, etc). E.g., Positive: beautiful, wonderful, good, amazing, Negative: bad, poor, terrible, cost someone an arm and a leg

(idiom). They are instrumental for opinion mining Three main ways to compile such a list:

Manual approach: not a bad idea, only an one- time effort Corpus-based approaches Dictionary-based approaches

Important to note: Some opinion words are context independent. (eg, good) Some are context dependent. (eg, long)

MIAS - June 14, 2007 36

Corpus-based approaches

Rely on syntactic or co-occurrence patterns in large corpuses. (Hazivassiloglou and McKeown, ACL-97; Turney, ACL-02; Yu and Hazivassiloglou, EMNLP-03; Kanayama and Nasukawa, EMNLP-06; Ding and Liu, 2007)

Can find domain (not context) dependent orientations (positive, negative, or neutral).

(Turney, ACL-02) and (Yu and Hazivassiloglou, EMNLP-03) are similar. Assign opinion orientations (polarities) to words/phrases. (Yu and Hazivassiloglou, EMNLP-03) is different from (Turney,

ACL-02) in that using more seed words (rather than two) and using log-

likelihood ratio (rather than PMI).

MIAS - June 14, 2007 37

Corpus-based approaches (contd)

Use constraints (or conventions) on connectives to identify opinion words (Hazivassiloglou and McKeown, ACL-97; Kanayama and Nasukawa, EMNLP-06; Ding and Liu, SIGIR-07). E.g., Conjunction: conjoined adjectives usually have the same orientation

(Hazivassiloglou and McKeown, ACL-97). E.g., “This car is beautiful and spacious.” (conjunction)

AND, OR, BUT, EITHER-OR, and NEITHER-NOR have similar constraints

Learning using log-linear model: determine if two conjoined adjectives are of the same or

different orientations. Clustering: produce two sets of words: positive and negative

Corpus: 21 million word 1987 Wall Street Journal corpus.

MIAS - June 14, 2007 38

Dictionary-based approaches

Typically use WordNet’s synsets and hierarchies to acquire opinion words Start with a small seed set of opinion words Use the set to search for synonyms and antonyms in

WordNet (Hu and Liu, KDD-04; Kim and Hovy, COLING-04).

Manual inspection may be used afterward. Use additional information (e.g., glosses) from WordNet

(Andreevskaia and Bergler, EACL-06) and learning (Esuti and Sebastiani, CIKM-05).

Weakness of the approach: Do not find domain and/or context dependent opinion words, e.g., small, long, fast.

MIAS - June 14, 2007 39

Roadmap



MIAS - June 14, 2007 40

Feature-based opinion mining and summarization (Hu and Liu, KDD-04)

Again focus on reviews (easier to work in a concrete domain!)

Objective: find what reviewers (opinion holders) liked and disliked Product features and opinions on the features

Since the number of reviews on an object can be large, an opinion summary should be produced. Desirable to be a structured summary. Easy to visualize and to compare. Analogous to but different from multi-document

summarization.

MIAS - June 14, 2007 41

The tasks

Recall the three tasks in our model.

Task 1: Extracting object features that have been commented on in each review.

Task 2: Determining whether the opinions on the features are positive, negative or neutral.

Task 3: Grouping feature synonyms. Summary

Task 2 may not be needed depending on the format of reviews.

MIAS - June 14, 2007 42

Different review format

Format 1 - Pros, Cons and detailed review: The reviewer is asked to describe Pros and Cons separately and also write a detailed review. Epinions.com uses this format.

Format 2 - Pros and Cons: The reviewer is asked to describe Pros and Cons separately. C|net.com used to use this format.

Format 3 - free format: The reviewer can write freely, i.e., no separation of Pros and Cons. Amazon.com uses this format.

MIAS - June 14, 2007 43

Format 1

GREAT Camera., Jun 3, 2004

Reviewer: jprice174 from Atlanta, Ga.

I did a lot of research last year before I bought this camera... It kinda hurt to leave behind my beloved nikon 35mm SLR, but I was going to Italy, and I needed something smaller, and digital.

The pictures coming out of this camera are amazing. The 'auto' feature takes great pictures most of the time. And with digital, you're not wasting film if the picture doesn't come out.

Format 2

Format 3

MIAS - June 14, 2007 44

Feature-based Summary (Hu and Liu, KDD-04)

GREAT Camera., Jun 3, 2004 Reviewer: jprice174 from Atlanta, Ga.

I did a lot of research last year before I bought this camera... It kinda hurt to leave behind my beloved nikon 35mm SLR, but I was going to Italy, and I needed something smaller, and digital. The pictures coming out of this camera are amazing. The 'auto' feature takes great pictures most of the time. And with digital, you're not wasting film if the picture doesn't come out. …

….

Feature Based Summary:

Feature1: picturePositive: 12 The pictures coming out of this camera are

amazing. Overall this is a good camera with a really good

picture clarity.…Negative: 2 The pictures come out hazy if your hands shake

even for a moment during the entire process of taking a picture.

Focusing on a display rack about 20 feet away in a brightly lit room during day time, pictures produced by this camera were blurry and in a shade of orange.

Feature2: battery life…

MIAS - June 14, 2007 45

Visual summarization & comparison

Summary of reviews of Digital camera 1

Picture Battery Size Weight Zoom

+

_

Comparison of reviews of

Digital camera 1

Digital camera 2 _

+

MIAS - June 14, 2007 46

Feature extraction from Pros and Cons of Format 1 (Liu et al WWW-03; Hu and Liu, AAAI-CAAW-05)

Observation: Each sentence segment in Pros or Cons contains only one feature. Sentence segments can be separated by commas, periods, semi-colons, hyphens, ‘&’’s, ‘and’’s, ‘but’’s, etc.

Pros in Example 1 can be separated into 3 segments:great photos <photo>easy to use <use>very small <small> <size>

Cons can be separated into 2 segments:battery usage <battery>included memory is stingy <memory>

MIAS - June 14, 2007 47

Extraction using label sequential rules

Label sequential rules (LSR) are a special kind of sequential patterns, discovered from sequences.

LSR Mining is supervised (Liu’s Web mining book 2006). The training data set is a set of sequences, e.g.,

“Included memory is stingy”

is turned into a sequence with POS tags.

{included, VB}{memory, NN}{is, VB}{stingy, JJ}

then turned into {included, VB}{$feature, NN}{is, VB}{stingy, JJ}

MIAS - June 14, 2007 48

Using LSRs for extraction

Based on a set of training sequences, we can mine label sequential rules, e.g.,

{easy, JJ }{to}{*, VB} {easy, JJ}{to}{$feature, VB} [sup = 10%, conf = 95%]

Feature Extraction Only the right hand side of each rule is needed. The word in the sentence segment of a new review that

matches $feature is extracted. We need to deal with conflict resolution also (multiple

rules are applicable.

MIAS - June 14, 2007 49

Extraction of features of formats 2 and 3

Reviews of these formats are usually complete sentencese.g., “the pictures are very clear.” Explicit feature: picture

“It is small enough to fit easily in a coat pocket or purse.” Implicit feature: size

Extraction: Frequency based approach Frequent features Infrequent features

MIAS - June 14, 2007 50

Frequency based approach(Hu and Liu, KDD-04)

Frequent features: those features that have been talked about by many reviewers.

Use sequential pattern mining Why the frequency based approach?

Different reviewers tell different stories (irrelevant) When product features are discussed, the words that they

use converge. They are main features.

Sequential pattern mining finds frequent phrases. Froogle has an implementation of the approach (no POS restriction).

MIAS - June 14, 2007 51

Using part-of relationship and the Web(Popescu and Etzioni, EMNLP-05)

Improved (Hu and Liu, KDD-04) by removing those frequent noun phrases that may not be features: better precision (a small drop in recall).

It identifies part-of relationship Each noun phrase is given a pointwise mutual information score

between the phrase and part discriminators associated with the product class, e.g., a scanner class.

The part discriminators for the scanner class are, “of scanner”, “scanner has”, “scanner comes with”, etc, which are used to find components or parts of scanners by searching on the Web: the KnowItAll approach, (Etzioni et al, WWW-04).

MIAS - June 14, 2007 52

Infrequent features extraction

How to find the infrequent features? Observation: the same opinion word can be used to describe

different features and objects. “The pictures are absolutely amazing.” “The software that comes with it is amazing.”

Frequent features

Opinion words

Infrequent features

MIAS - June 14, 2007 53

Identify feature synonyms

Liu et al (WWW-05) made an attempt using only WordNet. Carenini et al (K-CAP-05) proposed a more sophisticated method

based on several similarity metrics, but it requires a taxonomy of features to be given. The system merges each discovered feature to a feature node in

the taxonomy. The similarity metrics are defined based on string similarity,

synonyms and other distances measured using WordNet. Experimental results based on digital camera and DVD reviews

show promising results. Many ideas in information integration are applicable.

MIAS - June 14, 2007 54

Identify opinion orientation on feature

For each feature, we identify the sentiment or opinion orientation expressed by a reviewer.

We work based on sentences, but also consider, A sentence may contain multiple features. Different features may have different opinions. E.g., The battery life and picture quality are great (+), but the view

founder is small (-). Almost all approaches make use of opinion words and phrases. But

note again: Some opinion words have context independent orientations, e.g.

great. Some other opinion words have context dependent orientations,

e.g., “small” Many ways to use them.

MIAS - June 14, 2007 55

Aggregation of opinion words (Hu and Liu, KDD-04; Ding and Liu, SIGIR-07)

Input: a pair (f, s), where f is a feature and s is a sentence that contains f.

Output: whether the opinion on f in s is positive, negative, or neutral. Two steps:

Step 1: split the sentence if needed based on BUT words (but, except that, etc).

Step 2: work on the segment sf containing f. Let the set of opinion words in sf be w1, .., wn. Sum up their orientations (1, -1, 0), and assign the orientation to (f, s) accordingly.

In (Ding and Liu, SIGIR-07), step 2 is changed to

with better results. wi.o is the opinion orientation of wi. d(wi, f) is the distance from f to wi.

n

ii

i

fwd

ow1 ),(

.

MIAS - June 14, 2007 56

Context dependent opinions

Popescu and Etzioni (2005) used constraints of connectives in (Hazivassiloglou and McKeown, ACL-

97), and some additional constraints, e.g., morphological relationships, synonymy and antonymy, and

relaxation labeling to propagate opinion orientations to words and features.

Ding and Liu (2007) used constraints of connectives both at intra-sentence and inter-sentence

levels, and additional constraints of, e.g., TOO, BUT, NEGATION.to directly assign opinions to (f, s) with good results (> 0.85 of F-score).

MIAS - June 14, 2007 57

Roadmap



MIAS - June 14, 2007 58

Extraction of Comparatives

Comparative sentence mining Identify comparative sentences Extract comparative relations from them

MIAS - June 14, 2007 59

Linguistic Perspective

Comparative sentences use morphemes like more/most, -er/-est, less/least, as than and as are used to make a `standard’ against

which an entire entity is compared

Limitations Limited coverage

“In market capital, Intel is way ahead of AMD.”

Non-comparatives with comparative words “In the context of speed, faster means better.”

MIAS - June 14, 2007 60

Types of Comparatives

Gradable Non-Equal Gradable: Relations of the type greater or

less than Keywords like better, ahead, beats, etc “Optics of camera A is better than that of camera B”

Equative: Relations of type equal to Keywords and phrases like equal to, same as, both, all “Camera A and camera B both come in 7MP”

Superlative: Relations of the type greater or less than all others

Keywords and phrases like best, most, better than all “Camera A is the cheapest camera available in the market.”

MIAS - June 14, 2007 61

Types of Comparatives: non-gradable

Non-gradable: Sentences that compare features of two or more objects, but do not grade them. Sentences which imply: Object A is similar to or different from Object B with

regard to some features Object A has feature F1, Object B has feature F2 Object A has feature F, but Object B does not have

MIAS - June 14, 2007 62

Comparative Relation: gradable

Definition: A gradable comparative relation captures the essence of gradable comparative sentence and is represented with the following:

(relationWord, features, entityS1, entityS2, type) relationWord: The keyword used to express a

comparative relation in a sentence. features: a set of features being compared. entityS1 and entityS2: Sets of entities being

compared. type: non-equal gradable, equative or superlative

MIAS - June 14, 2007 63

Examples: Comparative relations

“car X has better controls than car Y”(relationWord = better, features = controls, entityS1 = carX, entityS2 = carY, type = non-equal-gradable)

“car X and car Y have equal mileage”(relationWord = equal, features = mileage, entityS1 = carX, entityS2 = carY, type = equative)

“car X is cheaper than both car Y and car Z”(relationWord = cheaper, features = null, entityS1 = carX, entityS2 = {carY, carZ}, type = non-equal-gradable)

“company X produces a variety of cars, but still best cars come from company Y”(relationWord = best, features = cars, entityS1 = companyY, entityS2 = companyX, type = superlative)

MIAS - June 14, 2007 64

Tasks

Given a collection of evaluative textsTask 1: Identify comparative sentences

Task 2: Categorize different types of comparative sentences.

Task 3: Extract comparative relations from the sentences

MIAS - June 14, 2007 65

Identify comparative sentences

Keyword strategy An observation: Its is easy to find a small set of

keywords that covers almost all comparative sentences, i.e., with a very high recall and a reasonable precision

A list of 83 keywords used in comparative sentences compiled by (Jinal and Liu, Sigir-06) including

Words with POS tags of JJR, JJS, RBR, RBS POS tags are used as keyword instead of individual

words Exceptions: more, less, most, least

Other indicative word like beat, exceed, ahead, etc. Phrases like in the lead, on par with, etc.

MIAS - June 14, 2007 66

2-step learning strategy

Step 1: Extract sentences which contain at least one keyword (recall = 98%, precision = 32% on our data set of gradables)

Step 2: Use Naïve Bayes classifier to classify sentences into two classes Comparative and non-comparative Attributes: class sequential rules (CSRs) generated

from sentences in step 1

MIAS - June 14, 2007 67

1. Sequence data preparation Use words within a radius r of a keyword to form a sequence

(words are replaced with POS tags) …

2. CSR generation Use different minimum supports for different keywords 13 manual rules, which were hard to generate automatically

3. Learning using a NB classifier Use CSRs and manual rules as attributes to build a final

classifier

MIAS - June 14, 2007 68

Classify different types of comparatives

Classify comparative sentences into three types: non-equal gradable, equative and superlative SVM learner gives the best result Asstribute set is the set of keywords If the sentence has a particular keywords in the

attribute set, the corresponding value is 1, and 0 otherwise.

MIAS - June 14, 2007 69

Extraction of comparative relations

Assumptions There is only one relation in a sentence Entities and features are nominals

Adjectival comparatives Does not deal with adverbial comparatives

3 steps Sequence data generation Label sequential rule (LSR) generation Build a sequential cover/extractor from LSRs

MIAS - June 14, 2007 70

Sequence data generation

Label Set = {$entityS1, $entityS2, $feature} Three labels are used as pivots to generate sequences.

Radius of 4 for optimal results Following words are also added

Distance words = {l1, l2, l3, l4, r1, r2, r3, r4} Special words #start and #end are used to mark the

start and the end of a sentence.

MIAS - June 14, 2007 71

Sequence data generation example

The comparative sentence

“Canon/NNP has/VBZ better/JJR optics/NNS”has $entityS1 “Canon” and $feature “optics”

Sequences are: <{#start>{l1}{$entityS1, NNP){r1}{has, VBZ}{r2}{better,

JJR}{r3}{$Feature, NNS}{r4}{#end}> <{#start>{l4}{$entityS1, NNP){l3}{has, VBZ}{l2}{better,

JJR}{l1}{$Feature, NNS}{r1}{#end}>

MIAS - June 14, 2007 72

Build a sequential cover from LSRs

LSR: {*, NN}{VBZ} { $entityS1, NN}{VBZ} Select the LSR rule with the highest confidence. Replace

the matched elements in the sentences that satisfy the rules with the labels in the rule.

Recalculate the confidence of each remaining rule based on the modified data from step 1.

Repeat step 1 and 2 until no rule left with confidence higher than minconf value (they sued 90%)

MIAS - June 14, 2007 73

Experimental Results (Jindal and Liu, AAAI 06)

Identifying Gradable Comparative Sentences Precision = 82% and recall = 81%

Classification into three gradable types SVM gave accuracy of 96%

Extraction of comparative relations LSR: F-score = 72%

MIAS - June 14, 2007 74

Summary

Two types of evaluations Direct opinions: We studied

The problem abstraction Sentiment analysis at document level, sentence level and

feature level

Comparisons: Very hard problems, but very useful

The current techniques are still in their infancy. Industrial applications are coming up

MIAS - June 14, 2007 75

END

Manual and Automatic Subjectivity and Sentiment

Analysis

MIAS - June 14, 2007 77

Outline

Corpus Annotation Pure NLP

Lexicon development Recognizing Contextual Polarity in Phrase-Level

Sentiment Analysis

Applications Product review mining

MIAS - June 14, 2007 78

Corpus AnnotationWiebe, Wilson, Cardie. Language Resources and Evaluation 39 (1-2), 2005

MIAS - June 14, 2007 79

Overview

Fine-grained: expression-level rather than sentence or document level The photo quality was the best that I have seen in a camera. The photo quality was the best that I have seen in a camera.

Annotate expressions of opinions, evaluations, emotions material attributed to a source, but presented objectively

MIAS - June 14, 2007 80

Overview

Fine-grained: expression-level rather than sentence or document level The photo quality was the best that I have seen in a camera. The photo quality was the best that I have seen in a camera.

Annotate expressions of opinions, evaluations, emotions, beliefs material attributed to a source, but presented objectively

MIAS - June 14, 2007 81

Overview

Opinions, evaluations, emotions, speculations are private states.

They are expressed in language by subjective expressions.

Private state: state that is not open to objective observation or verification.

Quirk, Greenbaum, Leech, Svartvik (1985). A Comprehensive Grammar of the English Language.

MIAS - June 14, 2007 82

Overview

Focus on three ways private states are expressed in language

Direct subjective expressions Expressive subjective elements Objective speech events

MIAS - June 14, 2007 83

Direct Subjective Expressions

Direct mentions of private states

The United States fears a spill-over from the anti-terrorist campaign.

Private states expressed in speech events

“We foresaw electoral fraud but not daylight robbery,” Tsvangirai said.

MIAS - June 14, 2007 84

Expressive Subjective Elements [Banfield

1982]

“We foresaw electoral fraud but not daylight robbery,” Tsvangirai said

The part of the US human rights report about China is full of absurdities and fabrications

MIAS - June 14, 2007 85

Objective Speech Events

Material attributed to a source, but presented as objective fact

The government, it added, has amended the Pakistan Citizenship Act 10 of 1951 to enable women of Pakistani descent to claim Pakistani nationality for their children born to foreign husbands.

MIAS - June 14, 2007 86

MIAS - June 14, 2007 87

Nested Sources

“The report is full of absurdities,’’ Xirao-Nima said the next day.

MIAS - June 14, 2007 88

Nested Sources


(Writer)

MIAS - June 14, 2007 89

Nested Sources


(Writer, Xirao-Nima)

MIAS - June 14, 2007 90

Nested Sources


(Writer Xirao-Nima)(Writer Xirao-Nima)

MIAS - June 14, 2007 91

Nested Sources


(Writer Xirao-Nima)(Writer Xirao-Nima)

(Writer)

MIAS - June 14, 2007 92

“The report is full of absurdities,” Xirao-Nima said the next day.

Objective speech event anchor: the entire sentence source: <writer> implicit: true

Direct subjective anchor: said source: <writer, Xirao-Nima> intensity: high expression intensity: neutral attitude type: negative target: report

Expressive subjective element anchor: full of absurdities source: <writer, Xirao-Nima> intensity: high attitude type: negative

MIAS - June 14, 2007 93





MIAS - June 14, 2007 94





MIAS - June 14, 2007 95





MIAS - June 14, 2007 96





MIAS - June 14, 2007 97





MIAS - June 14, 2007 98

“The US fears a spill-over’’, said Xirao-Nima, a

professor of foreign affairs at the Central University

for Nationalities.

MIAS - June 14, 2007 99



for Nationalities.

(Writer)

MIAS - June 14, 2007 100



for Nationalities.

(writer, Xirao-Nima)

MIAS - June 14, 2007 101



for Nationalities.

(writer, Xirao-Nima, US)

MIAS - June 14, 2007 102



for Nationalities.

(writer, Xirao-Nima, US) (writer, Xirao-Nima)(Writer)

MIAS - June 14, 2007 103


Objective speech event anchor: said source: <writer, Xirao-Nima>

Direct subjective anchor: fears source: <writer, Xirao-Nima, US> intensity: medium expression intensity: medium attitude type: negative target: spill-over



for Nationalities.

MIAS - June 14, 2007 104

The report has been strongly criticized and condemned bymany countries.

MIAS - June 14, 2007 105


Direct subjective anchor: strongly criticized and condemned source: <writer, many-countries> intensity: high expression intensity: high attitude type: negative target: report

The report has been strongly criticized and condemned bymany countries.

MIAS - June 14, 2007 106

As usual, the US state Department published its annual report on human rights practices in world countries last Monday.

And as usual, the portion about China contains little truth and many absurdities, exaggerations and fabrications.

MIAS - June 14, 2007 107

Objective speech event anchor : the entire 1st sentence source : <writer> implicit : true

Direct subjective anchor : the entire 2nd sentence source : <writer> implicit : true intensity : high expression intensity : medium attitude type : negative target : report

As usual, the US state Department published its annual report on human rights practices in world countries last Monday.

And as usual, the portion about China contains little truth and many absurdities, exaggerations and fabrications.

Expressive subjective element anchor : little truth source : <writer> intensity : medium attitude type : negative

Expressive subjective element anchor : many absurdities, exaggerations, and fabrications source : <writer> intensity : medium attitude type : negative

Expressive subjective element anchor : And as usual source : <writer> intensity : low attitude type : negative

MIAS - June 14, 2007 108

Corpus

www.cs.pitt.edu/mqpa/databaserelease (version 2)

English language versions of articles from the world press (187 news sources)

Also includes contextual polarity annotations (later)

Themes of the instructions: No rules about how particular words should be annotated.

Don’t take expressions out of context and think about what they could mean, but judge them as they are used in that sentence.

MIAS - June 14, 2007 109

Agreement

Inter-annotator agreement studies performed on various aspects of the scheme

Kappa is a measure of the degree of nonrandom agreement between observers and/or measurements of a specific categorical variable

Kappa values range between .70 and .80

MIAS - June 14, 2007 110

Agreement

Two council street wardens who helped lift a 14-ton bus off an injured schoolboy are to be especially commended for their heroic actions.

Nathan Thomson and Neville Sharpe will receive citations from the mayor of Croydon later this month.

Two council street wardens who helped lift a 14-ton bus off an injured schoolboy are to be especially commended for their heroic actions.

Nathan Thomson and Neville Sharpe will receive citations from the mayor of Croydon later this month.

Annotator 1 Annotator 2

MIAS - June 14, 2007 111

Agreement

Inter-annotator agreement studies performed on various aspects of the scheme

Kappa is a measure of the degree of nonrandom agreement between observers and/or measurements of a specific categorical variable

Kappa values range between .70 and .80

MIAS - June 14, 2007 112

Outline



Sentiment Analysis Applications

Product review mining

MIAS - June 14, 2007 113

Who does lexicon development ?

Humans

Semi-automatic

Fully automatic

MIAS - June 14, 2007 114

What?

Find relevant words, phrases, patterns that can be used to express subjectivity

Determine the polarity of subjective expressions

MIAS - June 14, 2007 115

Words

Adjectives (e.g. Hatzivassiloglou & McKeown 1997, Wiebe 2000, Kamps & Marx 2002, Andreevskaia &

Bergler 2006)

positive: honest important mature large patient

Ron Paul is the only honest man in Washington. Kitchell’s writing is unbelievably mature and is only likely to

get better. To humour me my patient father agrees yet again to my

choice of film

MIAS - June 14, 2007 116

Words


Bergler 2006)

positive negative: harmful hypocritical inefficient insecure

It was a macabre and hypocritical circus. Why are they being so inefficient ?

subjective: curious, peculiar, odd, likely, probably

MIAS - June 14, 2007 117

Words


Bergler 2006)

positive negative subjective: curious, peculiar, odd, likely, probable

He spoke of Sue as his probable successor. The two species are likely to flower at different times.

MIAS - June 14, 2007 118

Other parts of speech (e.g. Turney & Littman 2003, Esuli & Sebastiani 2006)

Verbs positive: praise, love negative: blame, criticize subjective: predict

Nouns positive: pleasure, enjoyment negative: pain, criticism subjective: prediction

MIAS - June 14, 2007 119

Phrases

Phrases containing adjectives and adverbs (e.g. Turney 2002,

Takamura et al. 2007 )

positive: high intelligence, low cost negative: little variation, many troubles

MIAS - June 14, 2007 120

Patterns

Lexico-syntactic patterns (Riloff & Wiebe 2003) way with <np>: … to ever let China use force to have its

way with … expense of <np>: at the expense of the world’s securty

and stability underlined <dobj>: Jiang’s subdued tone … underlined

his desire to avoid disputes …

MIAS - June 14, 2007 121

How?

How do we identify subjective items?

MIAS - June 14, 2007 122

How?


Assume that contexts are coherent

MIAS - June 14, 2007 123

Conjunction

MIAS - June 14, 2007 124

Statistical association

If words of the same orientation like to co-occur together, then the presence of one makes the other more probable

Use statistical measures of association to capture this interdependence Mutual Information (Church & Hanks 1989)

MIAS - June 14, 2007 125

How?


Assume that contexts are coherent Assume that alternatives are similarly subjective

MIAS - June 14, 2007 126

How?


Assume that contexts are coherent Assume that alternatives are similarly subjective

MIAS - June 14, 2007 127

WordNet

MIAS - June 14, 2007 128

WordNet

MIAS - June 14, 2007 129

WordNet relations

MIAS - June 14, 2007 130

WordNet relations

MIAS - June 14, 2007 131

WordNet relations

MIAS - June 14, 2007 132

WordNet glosses

MIAS - June 14, 2007 133

WordNet examples

MIAS - June 14, 2007 134

How?


Assume that contexts are coherent Assume that alternatives are similarly subjective Take advantage of word meanings

MIAS - June 14, 2007 135

*We cause great leaders

Specific papers using these ideas

MIAS - June 14, 2007 137

Hatzivassiloglou & McKeown 1997

1. Build training set: label all adjectives with frequency > 20

Test agreement with human annotators

MIAS - June 14, 2007 138


1. Build training set: label all adj. with frequency > 20; test agreement with human annotators

2. Extract all conjoined adjectives

nice and comfortable

nice and scenic

MIAS - June 14, 2007 139


3. A supervised learning algorithm builds a graph of adjectives linked by the same or different semantic orientation

nice

handsome

terrible

comfortable

painful

expensive

fun

scenic

MIAS - June 14, 2007 140


4. A clustering algorithm partitions the adjectives into two subsets

nice

handsome

terrible

comfortable

painful

expensive

fun

scenicslow

+

MIAS - June 14, 2007 141

Wiebe 2000Learning Subjective Adjectives From Corpora

Learning evaluation and opinion clues

Distributional similarity process, based on manual annotation

Refinement with lexical features Improved results from both

MIAS - June 14, 2007 142

Lin’s (1998) Distributional Similarity

I have a brown dogsubj

mod

obj

det

Word R W I subj havehave obj dogbrown mod dog . . .

MIAS - June 14, 2007 143

Lin’s Distributional Similarity

R W R W R W R WR W R W R W R W R W R W R W R W

Word1 Word2

R={subj, obj, etc.}

MIAS - June 14, 2007 144

Bizarre

strange similar scary unusual fascinatinginteresting curious tragic different contradictory peculiar silly sad absurdpoignant crazy funny comic compellingodd

MIAS - June 14, 2007 145

Experiments

910

MIAS - June 14, 2007 146

Experiments

Seeds Separatecorpus

Distributionalsimilarity

910

Seeds +SimilarWords

MIAS - June 14, 2007 147

Experiments

Seeds Separatecorpus

Distributionalsimilarity

Seeds +SimilarWords9

10S

Words

p(subjective | s)

S > Adj > Majority

MIAS - June 14, 2007 148

Turney 2002a,b

Determine the semantic orientation of each extracted phrase based on their association with seven positive and seven negative words

)()(

)&(log),(

21

21221 wordpwordp

wordwordpwordwordPMI

)_()_(

)_()_(log)( 2 queryphitsquerynNEARwordhits

querynhitsquerypNEARwordhitswordIRPMISO

MIAS - June 14, 2007 149

Turney 2002a,b

Determine the semantic orientation of each extracted phrase based on their association with seven positive and seven negative words

)()(

)&(log),(

21

21221 wordpwordp

wordwordpwordwordPMI

)_()_(

)_()_(log)( 2 queryphitsquerynNEARwordhits

querynhitsquerypNEARwordhitswordIRPMISO

MIAS - June 14, 2007 151

Riloff & Wiebe 2003

Observation: subjectivity comes in many (low-frequency) forms better to have more data

Boot-strapping produces cheap data High-precision classifiers label sentences as subjective or objective Extraction pattern learner gathers patterns biased towards

subjective texts Learned patterns are fed back into high precision classifier

MIAS - June 14, 2007 152

MIAS - June 14, 2007 153

Riloff & Wiebe 2003

Observation: subjectivity comes in many (low-frequency) forms better to have more data

Boot-strapping produces cheap data High-precision classifiers label sentences as subjective or objective Extraction pattern learner gathers patterns biased towards

subjective texts Learned patterns are fed back into high precision classifier

MIAS - June 14, 2007 154

Yu & Hatzivassiloglou 2003Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences

Classifying documents: naïve bayes, words as features Finding opinion sentences:

2 similarity approaches Naïve bayes (n-grams, POS, counts of polar words,

counts of polar sequences, average orientation) Multiple naïve bayes

MIAS - June 14, 2007 155

Yu & Hatzivassiloglou 2003Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinino Sentences

Tagging words and sentences: modified log-likelihood ratio of collocation with pos,

neg adjectives in seed sets Adjectives, adverbs, and verbs provide best

combination for tagging polarity of sentences

MIAS - June 14, 2007 157

Kim & Hovy 2005Automatic Detection of Opinion Bearing Words and Sentences

WordNet-based method for collecting opinion-bearing adjectives and verbs manually constructed strong seed set manually labeled reference sets (opinion-bearing or

not) for synonyms/antonyms of seed set, calculate an

opinion strength relative to reference sets expand further with naïve bayes classifier

MIAS - June 14, 2007 159

Kim & Hovy 2005Automatic Detection of Opinion Bearing Words and Sentences

Corpus-based method (WSJ) Calculate bias of words for particular text genre

(Editorials and Letter to editor)

)(Pr

)(Pr)(

#rob(w)EditorialP

wobalNoneditori

wobEditorialwScore

documentseditorialinwordstotal

documentseditorialinw

MIAS - June 14, 2007 161

Esuli & Sebastiani 2005Determining the semantic orientation of termsthrough gloss analysis

use final sets as gold standard to train a classifier, which uses all or part of the glosses in some format as features

the trained classifier can then be used to label any term that has a gloss with sentiment

w(awful) … w(dire) w(direful) … w(dread) W(dreaded) … …

MIAS - June 14, 2007 162

Esuli & Sebastiani 2006Determining Term Subjectivity and Term Orientation for Opinion Mining

Uses best system of 2005 paper Additional goal of distinguishing neutral from

positive/negative Multiple variations on learning approach, learner, training

set, feature selection The new problem is harder! Their best accuracy is 66%

(83% in 2005 paper)

MIAS - June 14, 2007 163

Suzuki et al. 2006Application of Semi-supervised learning to evaluative expression classification

Automatically extract and filter “evaluative expressions": The storage capacity of this HDD is high.

Classifies these as pos, neg, or neutral Use bootstrapping to be able to train an evaluative expression classifier

based on a larger collection of unlabeled data. Learn contexts that contain evaluative expressions

I am really happy because [the storage capacity is high] Unfortunately, [the laptop was too expensive].

MIAS - June 14, 2007 164

Suzuki et al. 2006


Classifies these as pos, neg, or neutral Use bootstrapping to be able to train an evaluative expression classifier

based on a larger collection of unlabeled data. Learn contexts that contain evaluative expressions


Evaluation

Attribute

Subject

MIAS - June 14, 2007 165

Suzuki et al. 2006


Classifies these as pos, neg, or neutral Use bootstrapping to be able to train an evaluative expression

classifier based on a larger collection of unlabeled data. Learn contexts that contain evaluative expressions


MIAS - June 14, 2007 166

Suzuki et al. 2006

Comparison of semi-supervised methods: Nigam et al.’s (2000) Naive Baiyes + EM method Naive Bayes + EM + SVM (SVM combined with Naive

Bayes + EM using Fisher kernel) And supervised methods:

Naive Bayes SVM

MIAS - June 14, 2007 167

Suzuki et al. 2006

Features: ‘Phew, [the noise] of [this HDD] is annoyingly high :-(’.

Candidate evaluative expression “Exclamation words” detected by POS tagger Emoticons and their emotional categories Words modifying words in the candidate evaluation expression Words modified by words in the candidate evaluative word

MIAS - June 14, 2007 168

Both Naive Bayes + EM, and Naive Bayes + EM + SVM work better than Naive Bayes and SVM.

Results show that Naive Bayes + EM boosted accuracy regardless of size of labeled data

Using more unlabeled data appeared to give better results. Qualitative analysis of the impact of the semi-supervised approaches by

looking at the top 100 features that had the highest probability P(feature—positive) before and after EM: more contextual features like exclamations, the happy emoticons, a

negation + ’but’, ‘therefore’ + ’interesting’, and ‘therefore‘ + ’comfortable.’

MIAS - June 14, 2007 169

Surely

… we’ve thought of everything by now?

MIAS - June 14, 2007 170

Word senses

Senses

MIAS - June 14, 2007 171

Senses

MIAS - June 14, 2007 172

Non-subjective senses of brilliant

1. Method for identifying brilliant material in paint - US Patent 7035464

2. Halley shines in a brilliant light.

3. In a classic pasodoble, an opening section in the minor mode features a brilliant trumpet melody, while the second section in the relative major begins with the violins.

MIAS - June 14, 2007 173

Andreevskaia and Bergler 2006Mining WordNet for Fuzzy Sentiment: Sentiment Tag Extraction from WordNet Glosses

Using wordnet relations (synonymy, antonymy and hyponymy) and glosses

Classify as positive, negative, or neutral Step algorithm with known seeds:

First expand with relations Next expand via glosses Filter out wrong POS and multiply assigned

Evaluate against General inquirer (which contains words, not word senses)

MIAS - June 14, 2007 175

Andreevskaia and Bergler 2006Mining WordNet for Fuzzy Sentiment: Sentiment Tag Extraction from WordNet Glosses

Disagreements between human labelers as a sign of fuzzy category structure HM and General Inquirer have 78.7% tag agreement

for shared adjectives Find way to measure the degree of centrality of words to

the category of sentiment Net overlap scores correlate with human agreement

MIAS - June 14, 2007 176

Outline





MIAS - June 14, 2007 177

Recognizing Contextual Polarity in Phrase-Level Sentiment AnalysisWilson, Wiebe, Hoffmann HLT-EMNLP-2005

MIAS - June 14, 2007 178

Most approaches use a lexicon of positive and negative wordsPrior polarity: out of context, positive or negative beautiful positive horrid negative

A word may appear in a phrase that expresses a different polarity in context

Contextual polarity

“Cheers to Timothy Whitfield for the wonderfully horrid visuals.”

Prior Polarity versus Contextual Polarity

MIAS - June 14, 2007 179

Example

Philip Clap, President of the National Environment Trust, sums up well the general thrust of the reaction of environmental movements: there is no reason at all to believe that the polluters are suddenly going to become reasonable.

MIAS - June 14, 2007 180

Example


MIAS - June 14, 2007 181


Example

prior polarityprior polarityContextual Contextual polaritypolarity

MIAS - June 14, 2007 182

Goal of This Work

Automatically distinguish prior and contextual polarity

MIAS - June 14, 2007 183

Approach

Use machine learning and variety of features

Achieve significant results for a large subset of sentiment expressions

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2

AllInstances

PolarInstances

MIAS - June 14, 2007 184

Outline

Introduction Manual Annotations and Corpus

Prior-Polarity Subjectivity Lexicon Experiments Conclusions

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 185

Manual Annotations

Subjective expressions of the MPQA corpus annotated with contextual polarity

MIAS - June 14, 2007 186

Annotation Scheme

Mark polarity of subjective expressions as positive, negative, both, or neutral

African observers generally approved of his victory while Western governments denounced it.

Besides, politicians refer to good and evil …

Jerome says the hospital feels no different than a hospital in the states.

positive

negative

both

neutral

MIAS - June 14, 2007 187

Annotation Scheme

Judge the contextual polarity of sentiment ultimately being conveyed

They have not succeeded, and will never succeed, in breaking the will of this valiant people.

MIAS - June 14, 2007 188

Annotation Scheme



MIAS - June 14, 2007 189

Annotation Scheme



MIAS - June 14, 2007 190

Annotation Scheme



MIAS - June 14, 2007 191

Corpus

425 documents from MPQA Opinion Corpus 15,991 subjective expressions in 8,984 sentences

Divided into two sets Development set for feature development

66 docs / 2,808 subjective expressions Experiment set

359 docs / 13,183 subjective expressions Divided into 10 folds for cross-validation

MIAS - June 14, 2007 192

Outline

Introduction Manual Annotations and Corpus Prior-Polarity Subjectivity Lexicon

Experiments Conclusions

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 193

Prior-Polarity Subjectivity Lexicon

Over 8,000 words from a variety of sources Both manually and automatically identified Positive/negative words from General Inquirer and

Hatzivassiloglou and McKeown (1997) All words in lexicon tagged with:

Prior polarity: positive, negative, both, neutral Reliability: strongly subjective (strongsubj),

weakly subjective (weaksubj)

MIAS - June 14, 2007 194

Outline

Introduction Manual Annotations Corpus Prior-Polarity Subjectivity Lexicon Experiments Conclusions

MIAS - June 14, 2007 195

Experiments

Give each instance its own label

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

Both Steps: BoosTexter AdaBoost.HM 5000 rounds boosting 10-fold cross validation

MIAS - June 14, 2007 196

Definition of Gold Standard

Given an instance inst from the lexicon:

if inst not in a subjective expression:

goldclass(inst) = neutral

else if inst in at least one positive and one negative subjective expression:

goldclass(inst) = both

else if inst in a mixture of negative and neutral:

goldclass(inst) = negative

else if inst in a mixture of positive and neutral:

goldclass(inst) = positive

else: goldclass(inst) = contextual polarity of subjective expression

MIAS - June 14, 2007 197

Features

Many inspired by Polanyi & Zaenen (2004): Contextual Valence Shifters

Example: little threat

little truth Others capture dependency relationships between words

Example:

wonderfully horrid

pos

mod

MIAS - June 14, 2007 198

1. Word features

2. Modification features

3. Structure features

4. Sentence features

5. Document feature

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 199

1. Word features2. Modification features3. Structure features4. Sentence features5. Document feature

Word token terrifies

Word part-of-speechVB

Context that terrifies me Prior Polarity

negative Reliability

strongsubj

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 200

1. Word features




5. Document feature

Binary features: Preceded by

adjective adverb (other than not) intensifier

Self intensifier Modifies

strongsubj clue weaksubj clue

Modified by strongsubj clue weaksubj clue

Dependency Parse Tree

The human rights

report

poses

a substantial

challenge

…

detadj mod adj

det

subj obj

p

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 201

1. Word features




5. Document feature

Binary features: In subject [The human rights report]

poses In copular I am confident In passive voice

must be regarded

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

The human rights

report

poses

a substantial

challenge

…

detadj mod adj

det

subj obj

p

MIAS - June 14, 2007 202

1. Word features2. Modification features3. Structure features4. Sentence features5. Document feature

Count of strongsubj clues in previous, current, next sentence

Count of weaksubj clues in previous, current, next sentence

Counts of various parts of speech

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 203

1. Word features




5. Document feature

Document topic (15) economics health

Kyoto protocol presidential election in Zimbabwe

…

Example: The disease can be contracted if a person is bitten by a certain tick or if a person comes into contact with the blood of a congo fever sufferer.

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 204

75.9

63.4

82.1

40

50

60

70

80

90

Accuracy Polar F Neutral F

Word token

Word + Prior Polarity

All Features

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

Results 1a

MIAS - June 14, 2007 205

30

40

50

60

70

80

Polar Recall Polar Precision

Word token


All Features

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

Results 1b

MIAS - June 14, 2007 206

Step 2: Polarity Classification

Classes positive, negative, both, neutral

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

19,506 5,671

MIAS - June 14, 2007 207

Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 208


Word token terrifies

Word prior polarity negative

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 209


Binary features: Negated

For example: not good does not look very good not only good but amazing

Negated subject

No politically prudent Israeli could support either of them.

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 210


Modifies polarity

5 values: positive, negative, neutral, both, not mod

substantial: negative

Modified by polarity


challenge: positive

substantial (pos) challenge (neg)

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 211


Conjunction polarity


good: negative

good (pos) and evil (neg)

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 212


General polarity shifter

have few risks/rewards

Negative polarity shifter

lack of understanding

Positive polarity shifter abate the damage

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

MIAS - June 14, 2007 213

65.7 65.1

77.2

46.2

30

40

50

60

70

80

90

Accuracy Pos F Neg F Neutral F

Word token


All Features

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

Results 2a

MIAS - June 14, 2007 214

40

50

60

70

80

90

PosRecall

Pos Prec NegRecall

Neg Prec

Word token


All Features

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

Results 2b

MIAS - June 14, 2007 215

Ablation experiments removing features:

1. Negated, negated subject

2. Modifies polarity, modified by polarity

3. Conjunction polarity

4. General, negative, positive polarity shifters

Corpus

Lexicon

Neutralor

Polar?

Step 1

ContextualPolarity?

Step 2All

InstancesPolar

Instances

Combination of features is needed to achieve significant results over baseline

MIAS - June 14, 2007 217

Conclusions

Presented a two-step approach to phrase-level sentiment analysis

1. Determine if an expression is neutral or polar

2. Determines contextual polarity of the ones that are polar

Automatically identify the contextual polarity of a large subset of sentiment expression

MIAS - June 14, 2007 218

Discussion: Lexicon <> Context

Objective Subjective

#1 a bunch of #2

#1 condemn #2

headless chicken #1

#1,3,5,6,7 hot #2,4,8

MIAS - June 14, 2007 219

He is not the sharpest knife in the drawer. She is a few fries short of a Happy Meal.

Stephanie McMahon is the next Stalin.

You are no Jack Kennedy.

Whatever.

MIAS - June 14, 2007 220

That was utterly foolish. No one would say that John is smart.

My little brother could have told you that. Senator NN has voted with the president 95% of the

times.

MIAS - June 14, 2007 221

Outline





MIAS - June 14, 2007 222


MIAS - June 14, 2007 223


Goal: summarize a set of reviews Targeted opinion mining: topic is given Two levels:

Product Product and features

Typically done for pre-identified reviews but review identification may be necessary

MIAS - June 14, 2007 224

Laptop review 1

A Keeper Reviewed By: N.N. on 5/12/2007 Tech Level: average - Ownership: 1 week to 1 month Pros: Price/Value. XP OS NOT VISTA! Screen good even in

bright daylignt. Easy to access USB, lightweight. Cons: A bit slow - since we purchased this for vacation travel

(email & photos) speed is not a problem. Other Thoughts: Would like to have card slots for camera/PDA

cards. Wish we could afford two so we can have a "spare".

MIAS - June 14, 2007 225

Laptop review 1





MIAS - June 14, 2007 226

Laptop review 2

By N.N. (New York - USA) - See all my reviewsI was looking for a laptop for long time, doing search, comparing brands, technology, cost/benefits etc.... I should say that I am a normal user and this laptop satisfied all my expectations, the screen size is perfect, its very light, powerful, bright, lighter, elegant, delicate... But the only think that I regret is the Battery life, barely 2 hours... some times less... it is too short... this laptop for a flight trip is not good companion... Even the short battery life I can say that I am very happy with my Laptop VAIO and I consider that I did the best decision. I am sure that I did the best decision buying the SONY VAIO

MIAS - June 14, 2007 227

Laptop review 2

By N.N. (New York - USA) - See all my reviewsI was looking for a laptop for long time, doing search, comparing brands, technology, cost/benefits etc.... I should say that I am a normal user and this laptop satisfied all my expectations, the screen size is perfect, its very light, powerful, bright, lighter, elegant, delicate... But the only think that I regret is the Battery life, barely 2 hours... some times less... it is too short... this laptop for a flight trip is not good companion... Even the short battery life I can say that I am very happy with my Laptop VAIO and I consider that I did the best decision. I am sure that I did the best decision buying the SONY VAIO

MIAS - June 14, 2007 230

Some challenges

Available NLP tools have harder time with review data (misspellings, incomplete sentences)

Level of user experience (novice, …, prosumer) Various types and formats of reviews Additional buyer/owner narrative What rating to assume for unmentioned features? How to aggregate positive and negative evaluations? How to present results?

MIAS - June 14, 2007 231

Core problems of review mining

Finding product features Recognizing opinions

MIAS - June 14, 2007 232

Feature finding

Wide variety of linguistic expressions can evoke a product feature … you can't see the LCD very well in sunlight. … it is very difficult to see the LCD. … in the sun, the LCD screen is invisible It is very difficult to take pictures outside in the sun

with only the LCD screen.

MIAS - June 14, 2007 233

Opinions v. Polar facts

Some statements invite emotional appraisal but do not explicitly denote appraisal.

While such polar facts may in a particular context seem to have an obvious value, their evaluation may be very different in another one.

MIAS - June 14, 2007 234





MIAS - June 14, 2007 235

Use Coherence to resolve orientation of polar facts

Is a sentence framed by two positive sentences likely to also be positive?

Can context help settle the interpretation of inherently non-evaluative attributes (e.g. hot room v. hot water in a hotel context; Popescu & Etzioni 2005) ?

MIAS - June 14, 2007 236

Product-level review-classification Train Naïve Bayes classifier using a corpus of self-

tagged reviews available from major web sites (C|net, amazon)

Refine the classifier using the same corpus before evaluating it on sentences mined from broad web searches

Dave, Lawrence, Pennock 2003Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews

MIAS - June 14, 2007 237

Feature selection Substitution (statistical, linguistic)

I called Kodak I called Nikon I called Fuji

Backing off to wordnet synsets Stemming N-grams arbitrary-length substrings


I called COMPANY

MIAS - June 14, 2007 238

Feature selection Substitution (statistical, linguistic) Backing off to wordnet synsets

brilliant -> {brainy, brilliant, smart as a whip}

Stemming N-grams arbitrary-length substrings


MIAS - June 14, 2007 239

Feature selection Substitution (statistical, linguistic) Backing off to wordnet synsets Stemming

bought them buying them buy them

N-grams arbitrary-length substrings


buy them

MIAS - June 14, 2007 240


Backing off to wordnet synsets Stemming N-grams

last long enough too hard to

arbitrary-length substrings


MIAS - June 14, 2007 241


Backing off to wordnet synsets Stemming N-grams arbitrary-length substrings


MIAS - June 14, 2007 242

Laplace (add-one) smoothing was found to be best 2 types of test (1 balanced, 1 unbalanced)

SVM did better on Test 2 (balanced data) but not Test 1

Experiments with weighting features did not give better results


MIAS - June 14, 2007 243

Hu & Liu 2004Mining Opinion Features in Customer Reviews

Here: explicit product features only, expressed as nouns or compound nouns

Use association rule mining technique rather than symbolic or statistical approach to terminology

Extract associated items (item-sets) based on support (>1%)

MIAS - June 14, 2007 244


Feature pruning compactness

“I had searched for a digital camera for 3 months.” “This is the best digital camera on the market” “The camera does not have a digital zoom”

Redundancy manual ; manual mode; manual setting

MIAS - June 14, 2007 245


For sentences with frequent feature, extract nearby adjective as opinion

Based on opinion words, gather infrequent features (N, NP nearest to an opinion adjective)

The salesman was easy going and let me try all the models on display.

MIAS - June 14, 2007 246

Yi & Niblack 2005Sentiment mining in WebFountain

MIAS - June 14, 2007 247


Product feature terms are extracted heuristically, with high precision For all definite base noun phrases,

the NN the JJ NN the NN NN NN …

calculate a statistic based on likelihood ratio test

MIAS - June 14, 2007 249


Manually constructed Sentiment lexicon: excellent JJ + Pattern database: impress + PP(by; with)

Sentiment miner identifies the best fitting pattern for a sentence based on the parse

MIAS - June 14, 2007 250


Manually constructed Sentiment lexicon: excellent JJ + Pattern database: impress + PP(by; with)

Sentiment miner identifies the best fitting pattern for a sentence based on the parse

Sentiment is assigned to opinion target

MIAS - June 14, 2007 251


Discussion of hard cases: Sentences that are ambiguous out of context Cases that did not express a sentiment at all Sentences that were not about the product:

Need to associate opinion and target

MIAS - June 14, 2007 252

Summary

Subjectivity is common in language

MIAS - June 14, 2007 253

Summary

Subjectivity is common in language Recognizing it is useful in many NLP tasks

MIAS - June 14, 2007 254

Summary

Subjectivity is common in language Recognizing it is useful in many NLP tasks It comes in many forms and often is context-dependent

MIAS - June 14, 2007 255

Summary Subjectivity is common in language Recognizing it is useful in many NLP tasks It comes in many forms and often is context-dependent Contextual coherence and distributional similarity are

important linguistic notions in lexicon building A wide variety of features seem to be necessary for

opinion and polarity recognition

MIAS - June 14, 2007 256

Summary

Goals for the future: Fine resolution on opinions (expression level) Integrating the various attributes of subjective

expressions (opinion expressions, sources, targets, etc.)

Additional material

MIAS - June 14, 2007 258

Extensions to the annotation schemeWilson & Wiebe 2005. Annotating Attributions and Private States

I think people are happy because Chavez has fallen.

direct subjective span: are happy source: <writer, I, People> attitude:

inferred attitude span: are happy because Chavez has fallen type: neg sentiment intensity: medium target:

target span: Chavez has fallen

target span: Chavez

attitude span: are happy type: pos sentiment intensity: medium target:

direct subjective span: think source: <writer, I> attitude:

attitude span: think type: positive arguing intensity: medium target:

target span: people are happy because Chavez has fallen

MIAS - June 14, 2007 259

Work on the intensity of private states

Theresa Wilson, Janyce Wiebe and Rebecca Hwa (2006). Recognizing strong and weak opinion clauses. Computational Intelligence, 22 (2), pp. 73-99.

Theresa Wilson. 2007. Ph.D. Thesis. Fine-grained Subjectivity and Sentiment Analysis: Recognizing the Intensity, Polarity, and Attitudes of private states.

MIAS - June 14, 2007 260

James R. Martin and Peter R.R. White. 2005. The Language of Evaluation: The Appraisal Framework. An approach to evaluation that comes from within the

theory of systemic-functional grammar. Website on this theory maintained by P.R. White:

http://www.grammatics.com/appraisal/index.html

More work related to lexicon building

MIAS - June 14, 2007 262

Alina Andreevskaia and Sabine Bergler. 2006. Sentiment Tag Extraction from WordNet Glosses.

Janyce Wiebe and Rada Mihalcea. 2006. Word Sense and Subjectivity

Riloff, Patwardhan, Wiebe. 2006. Feature Subsumption for Opinion Analysis.

MIAS - June 14, 2007 263

Takamura et al. 2007Extracting Semantic Orientations of Phrases from Dictionary

Use a Potts model to categorize Adj+Noun phrases Targets ambiguous adjectives like low, high, small, large Connect two nouns, if one appears in gloss of other Nodes have orientation values (pos, neg, neu) and are

connected by same or different orientation links

MIAS - June 14, 2007 264

A Sample Lexical Network

costcost

lossloss

expenditureexpenditure sacrificesacrifice

WORD GLOSS

cost loss or sacrifice, expenditure

loss something lost

loselose

MIAS - June 14, 2007 265

Takamura et al 2007

index for node

set of seed words

state of node i

class label of seed word i

constants

Probabilistic Model on the Lexical Network (Potts model)

MIAS - June 14, 2007 266

Takamura et al. 2007

The state of a seed word becomes Neighboring nodes tend to have the same

label.

“low cost” = “low expenditure”

MIAS - June 14, 2007 267

Takamura et al. 2007

Manually labeled adj+noun data provide noun seeds of known orientation

The network assigns orientation to nouns not seen in training data

Further work on review mining

MIAS - June 14, 2007 269

Morinaga et. al. 2002. Mining Product Reputations on the Web

Kobayashi et al. 2004. Collecting Evaluative Expressions for Opinion Extraction

Hu & Liu. 2006. Opinion Feature Extraction Using Class Sequential Rules

MIAS - June 14, 2007 270

Popescu & Etzioni 2005

Report on a product review mining system that extracts and labels opinion expressions their attributes

They use the relaxation-labeling technique from computer vision to perform unsupervised classification satisfying local constraints (which they call neighborhood features)

The system tries to solve several classification problems (e.g. opinion and target finding) at the same time rather than separately.

Applications of Subjectivity and

Sentiment analysis not discussed earlier

MIAS - June 14, 2007 272

Question Answering

MIAS - June 14, 2007 273

Question Answering

Much work on Subjectivity & Sentiment has been motivated by QA.

Yu, H. & Hatzivassiloglou, V. (2003) Kim, S. & Hovy, E. (AAAI-Workshop 2005)

Some QA work has also indicated that making a subjective/objective distinction would be useful for the seemingly objective task of definitional QA

Lita et al. (2005)

MIAS - June 14, 2007 274

Question Answering

Making subjective/objective distinction has been showed to be useful in answering opinion based questions in news text Stoyanov et al.(2005)

Making finer grained distinction of subjective types ( Sentiment and Arguing) further improves the QA system ( world press and online discussion forums) Somasundaran et al. (2007)

MIAS - June 14, 2007 275

Information Extraction

Information Extraction has been used to learn subjective patterns

Wiebe and Riloff (2005) Subjectivity has been shown to improve IE

Riloff et al. (2005)

MIAS - June 14, 2007 276

Summarization

Opinion Summaries from documents have been created Stoyanov & Cardie (2006)

They combine fine grained opinions from the same source to create a source specific summary of opinion

Carenini et al.(IUI-2006) They summarize a large corpora of evaluative text about a single entity (product)

Different aspects of subjectivity analysis have been used to enhance summarizing systems.

Seki et al. (2005) Summarization based on user’s needs (benefits, positive/negative factors,

commentary, etc).

MIAS - June 14, 2007 277

Blog analysis

Analysis of sentiments on Blog posts Chesley et al.(2006)

Perform subjectivity and polarity classification on blog posts

Sentiment has been used for blog analysis Balog et al. (2006)

Discover irregularities in temporal mood patterns (fear, excitement, etc) appearing in a large corpus of blogs

Kale et al. (2007) Use link polarity information to model trust and influence in the blogosphere

Blog sentiment has been used in applications Mishne and Glance (2006)

Analyze Blog sentiments about movies and correlate it with its sales

MIAS - June 14, 2007 278

Human Computer Interaction

Affect sensing Liu et al. (2003)

Human Robot Interaction Tokuhisa & Terashima (2006)

Correlate enthusiasm levels in dialogs with subjective language for human robot interaction

MIAS - June 14, 2007 279

Visualization

Visualization of sentiments Gregory et al. (2007)

Visualize affect distribution in social media (blogs) for a topic.

Gamon et al. (2005) Visualize sentiment and its orientation for a topic from large

number of customer feedback texts.

MIAS - June 14, 2007 280

Trends & Buzz

Stock market Koppel & Shtrimberg(2004)

Correlate positive/negative news stories about publicly traded companies and the stock price changes

Market Intelligence from message boards, forums, blogs. Glance et al. (2005)

MIAS - June 14, 2007 281

Source and Target Finding

MIAS - June 14, 2007 282

Bethard et al. 2004Automatic Extraction of Opinion Propositions and their Holders

Find verbs that express opinions in propositional form, and their holders Still, Vista officials realize they’re relatively fortunate.

Modify algorithms developed in earlier work on semantic parsing to perform binary classification (opinion or not)

Use presence of subjectivity clues to identify opinionated uses of verbs

MIAS - June 14, 2007 283

Choi et al.2005Identifying sources of opinions with conditional random fields and extraction patterns

Treats source finding as a combined sequential tagging and information extraction task

IE patterns are high precision, lower recall Base CRF uses information about noun phrase semantics,

morphology, syntax IE patterns connect opinion words to sources Conditional Random Fields given IE features perform better than

CRFs alone

MIAS - June 14, 2007 284

Kim & Hovy 2006Extracting opinions, opinion holders, andtopics expressed in online news media text

Perform semantic role labeling (FrameNet) for a set of adjectives and verbs (pos, neg)

Map semantic roles to holder and target E.g. for Desiring frame: Experiencer->Holder

Train on FN data, test on FN data and on news sentences collected and annotated by authors’ associates

Precision is higher for topics, recall for holders

MIAS - June 14, 2007 285

Choi, Breck, Cardie 2006 Joint extraction of entities and relations for opinion reocgnition

Find direct expressions of opinions and their sources jointly

Uses sequence-tagging CRF classifiers for opinion expressions, sources, and potential link relations

Integer linear programming combines local knowledge and incorporates constraints

Performance better even on the individual tasks

MIAS - June 14, 2007 286

Further references on Source and Target Finding

Breck & Cardie 2004. Playing the Telephone Game: Determining the Hierarchical Structure of Perspective and Speech Expressions.

Bloom et al. 2007. Extracting Appraisal Expressions.

opinion mining

Documents

general search

opinion search liu

search system

general web search

opinion retrievalsearch

introduction facts

organization opinion

usergenerated contentplace