10/24/2011 www.insemtives.eu 1

Ten ways to make your semantic app addicted - REVISITED

Elena Simperl

Tutorial at the ISWC2011, Bonn, Germany

Executive summary • Many aspects of semantic content authoring naturally rely

on human contribution. • Motivating users to contribute is essential for semantic

technologies to reach critical mass and ensure sustainable growth.

• This tutorial is about

– Methods and techniques to study incentives and motivators applicable to semantic content authoring scenarios.

– How to implement the results of such studies through technology design, usability engineering, and game mechanics.

www.insemtives.eu 2

Incentives and motivators

• Motivation is the driving force that makes humans achieve their goals.

• Incentives are ‘rewards’ assigned by an external ‘judge’ to a performer for undertaking a specific task. – Common belief (among

economists): incentives can be translated into a sum of money for all practical purposes.

• Incentives can be related to both extrinsic and intrinsic motivations.

• Extrinsic motivation if task is considered boring, dangerous, useless, socially undesirable, dislikable by the performer.

• Intrinsic motivation is driven by an interest or enjoyment in the task itself.

Examples of applications

www.insemtives.eu 4

Extrinsic vs intrinsic motivations • Successful volunteer crowdsourcing is difficult

to predict or replicate. – Highly context-specific. – Not applicable to arbitrary tasks.

• Reward models often easier to study and control.* – Different models: pay-per-time, pay-per-unit, winner-

takes-it-all… – Not always easy to abstract from social aspects (free-

riding, social pressure…). – May undermine intrinsic motivation.

* in cases when performance can be reliably measured

Examples (ii)

Mason & Watts: Financial incentives and the performance of the crowds, HCOMP 2009.

Amazon‘s Mechanical Turk

• Types of tasks: transcription, classification, and content generation, data collection, image tagging, website feedback, usability tests.*

• Increasingly used by academia. • Vertical solutions built on top. • Research on extensions for complex tasks.

* http://behind-the-enemy-lines.blogspot.com/2010/10/what-tasks-are-posted-on-mechanical.html

Tasks amenable to crowdsourcing

• Tasks that are decomposable into simpler tasks that are easy to perform.

• Performance is measurable. • No specific skills or expertise are required.

Patterns of tasks* • Solving a task

– Generate answers – Find additional information – Improve, edit, fix

• Evaluating the results of a task – Vote for accept/reject – Vote up/down to rank

potentially correct answers – Vote best/top-n results

• Flow control – Split the task – Aggregate partial results

• Example: open-scale tasks in Mturk – Generate, then vote. – Introduce random noise to

identify potential issues in the second step

* „Managing Crowdsourced Human Computation“@WWW2011, Ipeirotis

Gene

rate

ans

wer

Label image

Vote

ans

wer

s

Correct or not?

Examples (iii)

www.insemtives.eu 10

What makes game mechanics successfull?*

• Accelerated feedback cycles. – Annual performance appraisals vs immediate feedback to

maintain engagement. • Clear goals and rules of play.

– Players feel empowered to achieve goals vs fuzzy, complex system of rules in real-world.

• Compelling narrative. – Gamification builds a narrative that engages players to

participate and achieve the goals of the activity.

• But in the end it’s about what task users want to get better at.

*http://www.gartner.com/it/page.jsp?id=1629214 Images from http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/ and http://www.hideandseek.net/wp-content/uploads/2010/10/gamification_badges.jpg

Guidelines • Focus on the actual goal and incentivize related

actions. – Write posts, create graphics, annotate pictures, reply

to customers in a given time… • Build a community around the intended actions.

– Reward helping each other in performing the task and interaction.

– Reward recruiting new contributors. • Reward repeated actions.

– Actions become part of the daily routine.

Image from http://t1.gstatic.com/images?q=tbn:ANd9GcSzWEQdtagJy6lxiR2focH2D01Wpz7dzAilDuPsWnL0i4GAHgnm_0hyw3upqw

What tasks can be gamified?* • Tasks that are decomposable into simpler

tasks, nested tasks. • Performance is measurable. • Obvious rewarding scheme. • Skills can be arranged in a smooth learning

curve.

*http://www.lostgarden.com/2008/06/what-actitivies-that-can-be-turned-into.html Image from http://www.powwownow.co.uk/blog/wp-content/uploads/2011/06/gamification.jpeg

What is different about semantic systems?

• It‘s still about the context of the actual application.

• User engagement with semantic tasks in order to – Ensure knowledge is

relevant and up-to-date. – People accept the new

solution and understand its benefits.

– Avoid cold-start problems. – Optimize maintenance

costs.

Tasks in knowledge engineering • Definition of vocabulary • Conceptualization

– Based on competency questions – Identifying instances, classes, attributes,

relationships

• Documentation – Labeling and definitions. – Localization

• Evaluation and quality assurance – Matching conceptualization to documentation

• Alignment • Validating the results of automatic methods

www.insemtives.eu 15

16

http://www.ontogame.org http://apps.facebook.com/ontogame

OntoGame API • API that provides several methods that are

shared by the OntoGame games, such as: – Different agreement types (e.g. selection

agreement). – Input matching (e.g. , majority). – Game modes (multi-player, single player). – Player reliability evaluation. – Player matching (e.g., finding the optimal

partner to play). – Resource (i.e., data needed for games)

management. – Creating semantic content.

• http://insemtives.svn.sourceforge.net/viewvc/insemtives/generic-gaming-toolkit

www.insemtives.eu 17 10/24/2011

OntoGame games

10/24/2011 www.insemtives.eu 18

Case studies

• Methods applied – Mechanism design. – Participatory design. – Games with a purpose. – Crowdsourcing via MTurk.

• Semantic content authoring scenarios – Extending and populating

an ontology. – Aligning two ontologies. – Annotation of text, media

and Web APIs.

Lessons learned • Approach is feasible for mainstream domains, where a

(large-enough) knowledge corpus is available. • Advertisement is important. • Game design vs useful content.

– Reusing well-kwown game paradigms. – Reusing game outcomes and integration in existing workflows

and tools.

• But, the approach is per design less applicable because – Knowledge-intensive tasks that are not easily nestable. – Repetitive tasks players‘ retention?

• Cost-benefit analysis.

Using Mechanical Turk for semantic content authoring

• Many design decisions similar to GWAPs. – But clear incentives structures. – How to reliably compare games and MTurk results?

• Automatic generation of HITs depending on the types of tasks and inputs.

• Integration in productive environments.

– Protégé plug-in for managing and using crowdsourcing results.

Outline of the tutorial Time Presentation 14:00 – 14:45

Human contributions in semantic content authoring

14:45 – 15:30

Case study: motivating employees to annotate enterprise content semantically at Telefonica

15:30 – 16:00

Coffee break

16:00 – 16:45

Case study: Crowdsourcing the annotation of dynamic Web content at seekda

16:45 – 17:30

Case study: Content tagging at MoonZoo and MyTinyPlanets

17:30 – 18:00

Ten ways to make your semantic app addicted - revisited www.insemtives.eu 22

10/24/2011 www.insemtives.eu 23

Realizing the Semantic Web by encouraging millions of end-users to

create semantic content.