KDD 2011 Research Poster

Content - Driven Trust Propagation FramworkV. G. Vinod Vydiswaran, ChengXiang Zhai, and Dan Roth

University of Illinois at Urbana-Champaign

Incorporating text in trust models

Model parametersCan you trust news stories?

Even reputed sources make mistakes. Not all claims made by a source is

equally trustworthy. Some claims are purposefully misleading. How to verify free-text claims?

Acknowledgments

This research was supported by the Multimodal Information Access and Synthesis (MIAS) Center at the University of Illinois at Urbana-Champaign, part of CCICADA, a DHS Science and Technology Center of Excellence, and grants from the Army Research Laboratory under agreement W911NF-09-2-0053.

Contact details

vgvinodv@illinois.edu, czhai@cs.uiuc.edu, danr@illinois.edu

Claim 1

Claim n

Claim 2

.

.

.

Evidence ClaimsSources

Web sources

Evidence passages

Claim sentences

Incorporates semantics in trust computation using evidence.

Claims need not be structured tuples – they can be free-text sentences.

Framework does not assume that accurate Information Extraction is available.

A source can have different trust profile for different claims – not all claims from a source get equal weight.

Advantages over traditional models

Traditional two-layer fact-finder models

Claim 1

Claim n

Claim 2

…

[Yin, et al., 2007; Pasternack & Roth, 2010]

1( )e

2( )e

3( )e

1( )c

1( )w

2( )w

3( )w

1w

3e

2e

1e

1c

3w

2w

1 1( , )e c

2 1( , )e c

3 1( , )e c2 2( , )w e

1 1( , )w e

3 3( , )w e

1 2( , )e e1 3( , )e e

Computed scores : Claim veracity : Evidence trust : Source trust

Influence factors : Evidence similarity : Relevance : Source - Evidence

influence

( )c( )e( )w

( , )e c( , )w e

1 2( , )e e

Iterative formulation

# Topic Retrieval Two-stage models

Our model

1 Healthcare 0.886 0.895 0.932

2 Obama administration 0.852 0.876 0.927

3 Bush administration 0.931 0.921 0.971

4 Democratic policy 0.894 0.769 0.922

5 Republican policy 0.774 0.848 0.936

6 Immigration 0.820 0.952 0.983

7 Gay rights 0.832 0.864 0.807

8 Corruption 0.874 0.841 0.941

9 Election reform 0.864 0.889 0.908

10 WikiLeaks 0.886 0.860 0.825

Average 0.861 0.869 0.915

+6.3% Relative+6.3% Relative

There is a need to determine the truth value of a claim. This value depends on its source as well as on evidence.

Evidence documents influence each other and have different relevance to claims.

We developed a trust propagation framework that associates relevant evidence to claims and sources.

Global analysis of this data, taking into account relations between the stories, their relevance and their sources allows us to make progress in determining trustworthiness values over sources and claims.

Experiments with news trustworthiness show promising results on incorporating evidence in trustworthiness computation and improving “credibility” of retrieved results.

Conclusions

Data characteristics

Experimental resultsD. Using trust model to boost evidence retrieval

C. Does it depend on news genres?

A. Computing trust scores and trusted sources for specific claim topics

B. Finding trustworthy news sources and news reporters

Model brings credible documents to the top of the result list

Improvement in NDCG scores statistically significant.

Model helps bring out the disparity in credibility of reporting on specific topics

Model scores show influence of both popularity and average rating of articles.

Specific news sources appear to be trusted more for specific news genres.

23,164 news articles from 23 genres collected from Politics category of NewsTrust.org

All news articles were rated by human volunteers based on journalistic principles

Scored in the range [1,5], mean 3.70 Investigative reports most trusted (4.10),

Advertisements least (2.43)

Veracity ofnews reporting

Trustworthiness of news stories

Credibility of news sources