Bridging the Motivation Gap for Individual Annotators: What Can We Learn From Photo Annotation Systems?

Tabin HasanDept. of Computer Science

University of Trento

1st Workshop on Incentives for the Semantic WebISWC 2008, Karlsruhe, October 26th, 2008

Anthony JamesonFBK-irstTrento

Contents:1. The Two Motivation Gaps2. Bridging the … [See the title]3. Implications for the Other Workshop Papers

• (With audience participation)

The Community Motivation Gap(common view)

Creating metadata for yourself Creating metadata for a community

Spread the work around Provide individual as well as community benefit Provide quick confirmation and benefit Make incremental contributions usable Show what contributions are needed

Show what the current user is especially qualified to provide Provide a fun game Minimize privacy concerns Publicize contributions

Common strategies:

The Individual Motivation Gap

Creating metadata for yourself Exploiting your own metadata

Creating metadata for yourself Creating metadata for a community

A Photo Annotation Interface

"Sri Lanka dusk photos” detected by PhotoCompas using contextual metadata. From Naaman et al. (2004)

Metadata:Quantity and

Quality

Algorithms

· Clustering of similar objects

· Classifying into known classes

· Improvement of clustering or classification (via learning) User Interface

· Facilitation of batch annotation

· Recommendation of metadata

· Enjoyable actions and feedback

· Minimization of tedious actions

· Easy checking· Final judgment by

user

User Input

Annotation:· Sorting· Labeling

Naturally occurring:· Relevance

feedback while searching

· Annotating while sharing

ExternalResources

· Existing related text· Already annotated

objects· Information about

contexts· Persons who can

provide judgments

Affordances ofSituations

· Data currently clustered or classified?

· Time and attention available for annotation?

· Naturally occurring user actions that yield metadata?

External Resources

"Sri Lanka dusk photos” detected by PhotoCompas using contextual metadata. From Naaman et al. (2004)

Metadata:Quantity and

Quality

Algorithms

· Clustering of similar objects

· Classifying into known classes

· Improvement of clustering or classification (via learning) User Interface

· Facilitation of batch annotation

· Recommendation of metadata

· Enjoyable actions and feedback

· Minimization of tedious actions

· Easy checking· Final judgment by

user

User Input

Annotation:· Sorting· Labeling

Naturally occurring:· Relevance

feedback while searching

· Annotating while sharing

ExternalResources

· Existing related text· Already annotated

objects· Information about

contexts· Persons who can

provide judgments

Affordances ofSituations

· Data currently clustered or classified?

· Time and attention available for annotation?

· Naturally occurring user actions that yield metadata?

Algorithms, User Interface

SAPHARI, from Suh and Bederson (2007)

Metadata:Quantity and

Quality

Algorithms

· Clustering of similar objects

· Classifying into known classes

· Improvement of clustering or classification (via learning) User Interface

· Facilitation of batch annotation

· Recommendation of metadata

· Enjoyable actions and feedback

· Minimization of tedious actions

· Easy checking· Final judgment by

user

User Input

Annotation:· Sorting· Labeling

Naturally occurring:· Relevance

feedback while searching

· Annotating while sharing

ExternalResources

· Existing related text· Already annotated

objects· Information about

contexts· Persons who can

provide judgments

Affordances ofSituations

· Data currently clustered or classified?

· Time and attention available for annotation?

· Naturally occurring user actions that yield metadata?

User Input, Affordances of Situations

ARIA: cf. Lieberman, Rosenzweig, and Singh (2001)

Tag Learning

Metadata:Quantity and

Quality

Algorithms

· Clustering of similar objects

· Classifying into known classes

· Improvement of clustering or classification (via learning) User Interface

· Facilitation of batch annotation

· Recommendation of metadata

· Enjoyable actions and feedback

· Minimization of tedious actions

· Easy checking· Final judgment by

user

User Input

Annotation:· Sorting· Labeling

Naturally occurring:· Relevance

feedback while searching

· Annotating while sharing

ExternalResources

· Existing related text· Already annotated

objects· Information about

contexts· Persons who can

provide judgments

Affordances ofSituations

· Data currently clustered or classified?

· Time and attention available for annotation?

· Naturally occurring user actions that yield metadata?

Learning tags from examples

Geographic and event data sources

Suggesting annotations of new photos

Correcting incorrect suggested annotations

Best when user is uploading many photos related to a given place / event

SOMNet

Metadata:Quantity and

Quality

Algorithms

· Clustering of similar objects

· Classifying into known classes

· Improvement of clustering or classification (via learning) User Interface

· Facilitation of batch annotation

· Recommendation of metadata

· Enjoyable actions and feedback

· Minimization of tedious actions

· Easy checking· Final judgment by

user

User Input

Annotation:· Sorting· Labeling

Naturally occurring:· Relevance

feedback while searching

· Annotating while sharing

ExternalResources

· Existing related text· Already annotated

objects· Information about

contexts· Persons who can

provide judgments

Affordances ofSituations

· Data currently clustered or classified?

· Time and attention available for annotation?

· Naturally occurring user actions that yield metadata?

Better support for choice of terms

Encourage case entry when doctor has been dealing with the relevant information

Use KB of previous cases to support autocompletion

Enable “scratching own itch” while browsing

Intelligently use relevant data in doctors’ own computers

Collaborative IR Augmentation

Metadata:Quantity and

Quality

Algorithms

· Clustering of similar objects

· Classifying into known classes

· Improvement of clustering or classification (via learning) User Interface

· Facilitation of batch annotation

· Recommendation of metadata

· Enjoyable actions and feedback

· Minimization of tedious actions

· Easy checking· Final judgment by

user

User Input

Annotation:· Sorting· Labeling

Naturally occurring:· Relevance

feedback while searching

· Annotating while sharing

ExternalResources

· Existing related text· Already annotated

objects· Information about

contexts· Persons who can

provide judgments

Affordances ofSituations

· Data currently clustered or classified?

· Time and attention available for annotation?

· Naturally occurring user actions that yield metadata?

Suggest mapping on basis of previous learning

Facilitate contribution after noticing of gap

Use machine learning to exploit KB of existing keyword queries and formal representations

Support testing and debugging to require minimal user effort

Constitution-Based Game

Metadata:Quantity and

Quality

Algorithms

· Clustering of similar objects

· Classifying into known classes

· Improvement of clustering or classification (via learning) User Interface

· Facilitation of batch annotation

· Recommendation of metadata

· Enjoyable actions and feedback

· Minimization of tedious actions

· Easy checking· Final judgment by

user

User Input

Annotation:· Sorting· Labeling

Naturally occurring:· Relevance

feedback while searching

· Annotating while sharing

ExternalResources

· Existing related text· Already annotated

objects· Information about

contexts· Persons who can

provide judgments

Affordances ofSituations

· Data currently clustered or classified?

· Time and attention available for annotation?

· Naturally occurring user actions that yield metadata?

Make sure that the game is actually fun!

Inverse Search

Metadata:Quantity and

Quality

Algorithms

· Clustering of similar objects

· Classifying into known classes

· Improvement of clustering or classification (via learning) User Interface

· Facilitation of batch annotation

· Recommendation of metadata

· Enjoyable actions and feedback

· Minimization of tedious actions

· Easy checking· Final judgment by

user

User Input

Annotation:· Sorting· Labeling

Naturally occurring:· Relevance

feedback while searching

· Annotating while sharing

ExternalResources

· Existing related text· Already annotated

objects· Information about

contexts· Persons who can

provide judgments

Affordances ofSituations

· Data currently clustered or classified?

· Time and attention available for annotation?

· Naturally occurring user actions that yield metadata?

KB of aggregated information needs

Support batch export of subsets of private data

Algorithms for recommending information to be made public

Facilitate export when private data is originally added

Community Motivation Gap• Spread the work around so that each person

needs to do only a bit– Collaborative IR augmentation

» "collaborative"

• Provide individual benefit as well as community benefit

– Collaborative IR augmentation» "immediate"

Community Motivation Gap

• Provide quick confirmation of contribution and quick benefit

– Collaborative IR augmentation» "immediate"

– SOMNet» People want to know their contributions are being

taken seriously

Community Motivation Gap

• Ensure that even incremental contributions can be utilized

– Collaborative IR augmentation» "incremental", "partial"

• Show what contributions are needed (and which ones the current user is especially qualified to provide)

– Inverse search

Community Motivation Gap

• Provide a fun game in which people do the desired work

– Constitution-based game

• Protect privacy– SOMNet

» People wanted to avoid revealing gaps in knowledge– Inverse search

» Encourage people to move knowledge from private to public when it's needed

• Publicize (and maybe publicly evaluate) contributions