learning similarity functions for topic detection in online reputation monitoring

Linear Support Vector Machine (SVM) High correlation with Maximal Pairwise Accuracy (theoretical upper bound) Results can be extrapolated to other Machine Learning algorithms Most signals are able to improve term-based features Supervised system consistently improves the performance of its unsupervised counterpart Regardless of how the similarity threshold is set Topic Detection as Clustering Task Conclusions Effect of Twitter Signals Effect of Training Process Learning Similarity Functions for Topic Detection in Online Reputation Monitoring SIGIR 2014 Workshop on Semantic Matching in Information Retrieval (SMIR) Gold Coast, Australia. July 11 th , 2014. Damiano Spina, Julio Gonzalo, Enrique Amigó {damiano,julio,enrique}@lsi.uned.es Acknowledgments. This research was partially supported by the European Community's FP7 Programme under grant agreement nr. 288024 (LiMoSINe), the Spanish Ministry of Education (FPU grant nr. AP2009-0507), the Spanish Ministry of Science and Innovation (Holopedia Project, TIN2010-21128-C02), the UNED (project nr. 2012V/PUNED/0004), the Regional Government of Madrid and the ESF under MA2VICMR (S2009/TIC-1542), a Google Faculty Research Award (Axiometrics Project) and a SIGIR Student Travel Grant. What Are People Saying About a Given Entity Right Now? Entity = Brand, Company, Organization, Public Figure Early Detection of topics that may damage the reputation of the entity Twitter’s Long Tail: Sparsity http://bit.ly/simFunctionsORM Learning Similarity Functions System AUC MAR Text only SVM(terms_jaccard)+HAC 0.40 0.59 All Twitter Signals SVM(all features)+HAC 0.41 0.61* System AUC MAR Unsupervised terms_jaccard+HAC 0.38 0.57 Supervised SVM(terms_jaccard)+HAC 0.40 0.59* RepLab@CLEF 2013 61 entities 100,869 tweets annotated with 8,765 different topics Training ≠ Test Topics, no overlap in time 544 (1,109) tweets and 57 (87) topics for training (testing) per entity Dataset for Topic Detection Task Evaluation of Online Reputation Monitoring Systems http://nlp.uned.es/replab2013 Twitter signals(authors, timestamps, hashtags , etc.) can indeed be used to improve topic detection w.r.t. textual content only It is possible to learn effectively from manually annotated topics (pairwise tweet similarity) Organizational (“Hate Opinions”) vs. Event-like Topics (“Director of the Bank Accused of Evading Taxes”) Learning Similarity Functions + Hierarchical Agglomerative Clustering (HAC) significantly outperforms the state-of-the-art (RepLab Topic Detection systems) It gets close to the inter-annotator agreement rate: 0.48 F 1 (R,S) When data is sparse (as in the ORM scenario) it can be more effective than using probabilistic generative models such as Temporal Twitter-LDA http://nlp.uned.es Previous annotations can be used to learn better topic models, although differences are not large

Upload: damiano-spina-valenti

Post on 07-Jul-2015

488 views

Category:

Science

3 download

Report

Download

Tags:

Embed Size (px):

DESCRIPTION

Poster presented at SIGIR 2014 Workshop on Semantic Mathing for Information Retrieval

TRANSCRIPT

Page 1: Learning Similarity Functions for Topic Detection in Online Reputation Monitoring

Linear Support Vector Machine (SVM) High correlation with Maximal Pairwise Accuracy (theoretical upper bound)

Results can be extrapolated to other Machine Learning algorithms

Most signals are able to improve term-based features

Supervised system consistently improves the performance of its unsupervised counterpart

Regardless of how the similarity threshold is set

Topic Detection as Clustering Task

Conclusions

Effect of Twitter Signals Effect of Training Process

Learning Similarity Functions for Topic Detection in Online Reputation Monitoring

SIGIR 2014 Workshop on Semantic Matching in Information Retrieval (SMIR) Gold Coast, Australia. July 11th, 2014.

Damiano Spina, Julio Gonzalo, Enrique Amigó {damiano,julio,enrique}@lsi.uned.es

Acknowledgments. This research was partially supported by the European Community's FP7 Programme under grant agreement nr. 288024 (LiMoSINe), the Spanish Ministry of Education (FPU grant nr. AP2009-0507), the Spanish Ministry of Science and Innovation (Holopedia Project, TIN2010-21128-C02), the UNED (project nr. 2012V/PUNED/0004), the Regional Government of Madrid and the ESF under MA2VICMR (S2009/TIC-1542), a Google Faculty Research Award (Axiometrics Project) and a SIGIR Student Travel Grant.

What Are People Saying About a Given Entity Right Now?

Entity = Brand, Company, Organization, Public Figure

Early Detection of topics that may damage the reputation of the entity

Twitter’s Long Tail: Sparsity

http://bit.ly/simFunctionsORM

Learning Similarity Functions

System AUC MAR

Text only SVM(terms_jaccard)+HAC

0.40 0.59

All Twitter Signals SVM(all features)+HAC

0.41 0.61*

System AUC MAR

Unsupervised terms_jaccard+HAC

0.38 0.57

Supervised SVM(terms_jaccard)+HAC

0.40 0.59*

RepLab@CLEF 2013

61 entities 100,869 tweets annotated with 8,765 different topics

Training ≠ Test Topics, no overlap in time

544 (1,109) tweets and 57 (87) topics for training (testing) per entity

Dataset for Topic Detection Task

Evaluation of Online Reputation Monitoring Systems http://nlp.uned.es/replab2013

Twitter signals(authors, timestamps, hashtags , etc.) can indeed be used to improve topic detection w.r.t. textual content only

It is possible to learn effectively from manually annotated topics (pairwise tweet similarity)

Organizational (“Hate Opinions”) vs. Event-like Topics (“Director of the Bank Accused of Evading Taxes”)

Learning Similarity Functions + Hierarchical Agglomerative Clustering (HAC) significantly outperforms the state-of-the-art (RepLab Topic Detection systems)

It gets close to the inter-annotator agreement rate: 0.48 F1(R,S) When data is sparse (as in the ORM scenario) it can be more effective than using probabilistic generative models such as Temporal Twitter-LDA

http://nlp.uned.es

Previous annotations can be used to learn better topic models, although differences are not large

Email Reputation: IP vs. Domain Reputation

REPUTATION IS BIGGER · BIGGER REPUTATION IS THAN BRAND . REPUTATION IS EVERYTHING . IS BIGGER THAN BRAND REPUTATION . BYPRODUCT OF INTERNET . DECISIONS ALMOST MADE BY MACHINE . REPUTATION

Illud: Utilizing Semantic Similarity for Image Search · Topic Model StaySense Cosine Similarity on Records in Topic Execution Pipeline Data Management Pipeline Search engines commonly

Reputation Premium and Reputation Management: Evidence ...mxiao/research/reputation_FJX.pdf · Reputation Premium and Reputation Management: Evidence from the Largest e ... reputation

Reputation Management - Monitor Your Online Reputation

DRAFT Module Guidance Document Geometry Module 2 · DRAFT Module Guidance Document Geometry Module 2 Topic A Scale Drawings 5 days Topic B Dilations 5 days Topic C Similarity and

Is the Reputation Quotient a valid and reliable measure ... · building. Purpose – This ... SME Small and Medium-sized Enterprise ... Reputation is without doubt a relevant topic

DRAFT: Opinion Space Pilot Study Topic: Reputation of the American Automotive Industry Participants: Automotive Industry Online Community Research Period:

1 Lesson 3.4.6 Congruence and Similarity Congruence and Similarity

Reputation and Status: Expanding the Role of Social ...€¦ · Reputation Firm 23.1% 28 Reputation Individual 5.8% 7 Reputation Dyadic 2.5% 3 Reputation Group or team 3.3% 4 Reputation