google tech talk: reconsidering relevance

© 2009 Endeca Technologies, Inc. All rights reserved.

Reconsidering Relevance

Daniel TunkelangChief Scientist, Endeca

© 2009 Endeca Technologies, Inc. All rights reserved.2

howdy!

• 1988 – 1992

• 1993 – 1998

• 1999 -


overview

what is relevance?

what’s wrong with relevance?

what are the alternatives?


but first let’s set the stage


iconic businesses of the 20th and 21st centuries

I’m Feeling Lucky


process and scale orchestration


but there’s a dark side


users are satisfied


an interesting contrast

“Search on the internet is solved. I always find what I need.

But why not in the enterprise?

Seems like a solution waiting to happen.”

- a Fortune 500 CTO


the real questions

• What is “search on the internet” and why is it perceived a solved problem?

• What is “search in the enterprise” and why is it perceived as an unsolved problem?

• And what does this have to do with relevance?


easy vs. hard search problems

• easywhere to buy Ender in Exile?

• hardgood novel to read on the beach?

• easyproof that sorting has n log n lower bound?

• hardalgorithm to sort partially ordered set, given a constant-time comparator?


what is relevance?




defining relevance

Relevance is defined as a measure of information conveyed by a document relative to a query.

It is shown that the relationship between the document and the query, though necessary, is not sufficient to determine relevance.

William Goffman, On relevance as a measure, 1964.


we need more definitions


let’s work top-down

• information retrieval (IR) =

study of retrieval of information (not data) from collection of written documents

retrieved documents aim at satisfying user information need


IR assumes information needs

• user information need =

natural language declaration of informational need of user

• query =

expression of user information need in input language provided by information system


relevance drives IR modeling

• modeling =

studies algorithms used for ranking documents according to system assigned likelihood of relevance

• model =

a set of premises and an algorithm for ranking documents with regard to a user query


a relevance-centric approach

information Need query select from results

rank using IR model

USER:

SYSTEM:tf-idf PageRank


what is relevance?




our first communication problem

information need query

• 2 words?• natural language?• telepathy?


and the game of telephone continues

query rank using IR model

• cumulative error• relevance is subjective• what Goffman said


and hopefully users feel lucky

rank using IR model

• selection bias• inefficient channel• backup plan?

select from results


queries are misinterpreted

Results 1-10 out of about 344,000,000 for ir


ranked lists are inefficient


assumptions of relevance-centric approach

• self-awareness

• self-expression

• model knows best

• answer is a document

• one-shot query


can we do better?


what is relevance?




human-computer information retrieval

• don’t just guess the user’s intent– optimize communication

• increase user responsibility and control– require and reward human intellectual effort

“Toward Human-Computer Information Retrieval”

Gary Marchionini


human computer information retrieval


a concrete use case

• Colleague:

Hey Daniel! You should check out what this guy Steve Pollitt’s been researching. Sounds right up your alley.

• Daniel:

Sure thing, I’ll look into it.


google him!


google scholar him?


rexa him?


getting better


hcir-inspired interface


tags provide summarization and guidance


my information need evolves as i learn


hcir – implementing the vision


scatter/gather: a search for “star”


faceted search


practical considerations

• which facets to show

• which facet values to show

• when to suggest faceted refinement

• how to automate faceted classification


showing the right facets: microwaves


showing the right facets: ceiling fans


query-driven clarification before refinement

Matching Categories include:

Appliances > Small Appliances > Irons & Steamers

Appliances > Small Appliances > Microwaves & Steamers

Bath > Sauna & Spas > Steamers

Kitchen > Bakeware & Cookware > Cookware >Open Stock Pots > Double Boilers & Steamers

Kitchen > Small Appliances > Steamers


results-driven clarification before refinement

Search: storage


crowd-sourcing to tag documents


recall

precision

hcir cheats the precision / recall trade-off


set retrieval 2.0

• set retrieval that responds to queries with– overview of the user's current context– organized set of options for exploration

• contextual summaries of document sets– optimize system’s communication with user

• query refinement options– optimize user’s communication with system


hcir using set retrieval 2.0

emphasize set summaries over ranked lists

establish a dialog between the user and the data

enable exploration and discovery


think outside the (search) box

• relevance-centric search solves many use cases

• but not some of the most valuable ones

• support interaction, exploration

• human-computer information retrieval


one more thing

…


“Google's mission is to organize the

world's information and make it

universally accessible and useful.”


organizer or referee?


thank you

communication 1.0email: [email protected]

communication 2.0blog: http://thenoisychannel.com

twitter: http://twitter.com/dtunkelang

google tech talk: reconsidering relevance

Technology