insemtives stanford
DESCRIPTION
TRANSCRIPT
8/26/2011 www.insemtives.eu 1
Incentives and motivators for collaborative knowledge creation
Elena SimperlTalk at the Stanford Center for Biomedical Information, Stanford, CA
Insemtives in a nutshell• Many aspects of semantic content authoring naturally rely on human
contribution.
• Motivating users to contribute is essential for semantic technologies to reach critical mass and ensure sustainable growth.
• Insemtives works on – Best practices and guidelines for incentives-compatible technology design.– Enabling technology to realize incentivized semantic applications.– Showcased in three case studies: enterprise knowledge management;
services marketplace; multimedia management within virtual worlds.
www.insemtives.eu 2
The approach
• Typology of semantic content authoring tasks and theways people could be motivated to address them.
• Procedural ordering of methods and techniques to study incentives and motivators applicable to semantic content authoring scenarios.
• Guidelines and best practices for the implementation of the results of such studies through participatory design, usability engineering, and mechanism design.
• Pilots, showcases and enabling technology.
www.insemtives.eu 3
Incentives and motivators
• Motivation is the driving force that makes humans achieve their goals.
• Incentives are ‘rewards’ assigned by an external ‘judge’ to a performer for undertaking a specific task.– Common belief (among
economists): incentives can be translated into a sum of money for all practical purposes.
• Incentives can be related to both extrinsic and intrinsic motivations.
• Extrinsic motivation if task is considered boring, dangerous, useless, socially undesirable, dislikable by the performer.
• Intrinsic motivation is driven by an interest or enjoyment in the task itself.
Examples of applications
www.insemtives.eu 5
Extrinsic vs intrinsic motivations• Successful volunteer crowdsourcing is difficult
to predict or replicate. – Highly context-specific.– Not applicable to arbitrary tasks.
• Reward models often easier to study and control.*– Different models: pay-per-time, pay-per-unit, winner-
takes-it-all…– Not always easy to abstract from social aspects (free-
riding, social pressure…).– May undermine intrinsic motivation.
* in cases when performance can be reliably measured
Examples (ii)
Mason & Watts: Financial incentives and the performance of the crowds, HCOMP 2009.
Which tasks can be crowdsourcedand how?
• Modularity/Divisibility: can the task be divided into smaller chunks? How complex is the control flow? How can (intermediary) results be evaluated?– Casual games– Amazon’s Mturk– (Software development)
• Skills and expertise: doesthe task address a broad oran expert audience? – CAPTCHAs– Casual games
• Combinability: group performance– Additive: pulling a rope
(group performs better than individuals, but each individual pulls less hard)
– Conjunctive: running in a pack (performance is that of the weakest member, group size reduces group performance)
– Disjunctive: answering a quiz (group size increases group performance in term of the time needed to answer)
www.insemtives.eu 8
Amazon‘s Mechanical Turk
• Types of tasks: transcription, classification, and contentgeneration, data collection, image tagging, website feedback, usability tests.*
• Increasingly used by academia.• Vertical solutions built on top.• Research on extensions for complex tasks.
* http://behind-the-enemy-lines.blogspot.com/2010/10/what-tasks-are-posted-on-mechanical.html
Patterns of tasks*• Solving a task
– Generate answers– Find additional information– Improve, edit, fix
• Evaluating the results of a task– Vote for accept/reject– Vote up/down to rank
potentially correct answers– Vote best/top-n results
• Flow control– Split the task– Aggregate partial results
• Example: open-scale tasksin Mturk– Generate, then vote.– Introduce random noise to
identify potential issues in the second step
* „Managing Crowdsourced Human Computation“@WWW2011, Ipeirotis
Gene
rate
answ
er Label image
Vote
answ
ers
Corrector not?
Examples (iii)
www.insemtives.eu 11
Gamification features*
• Accelerated feedback cycles. – Annual performance appraisals vs immediate
feedback to maintain engagement.
• Clear goals and rules of play. – Players feel empowered to achieve goals vs fuzzy,
complex system of rules in real-world.
• Compelling narrative. – Gamification builds a narrative that engages players to
participate and achieve the goals of the activity.
*http://www.gartner.com/it/page.jsp?id=1629214
What tasks can be gamified?*
• Decomposable into simpler tasks.
• Nested tasks.• Performance is
measurable.• Obvious rewarding
scheme.• Skills can be arranged in
a smooth learning curve.
*http://www.lostgarden.com/2008/06/what-actitivies-that-can-be-turned-into.htmlImage from http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/
What is different about semantic systems?
• Semantic Web toolsvs applications. – Intelligent (specialized)
Web sites (portals) with improved (local) search based on vocabularies and ontologies.
– X2X integration (often combined with Web services).
– Knowledge representation, communication and exchange.
What do you want your users to do?
• Semantic applications– Context of the actual application.– Need to involve users in knowledge acquisition and
engineering tasks?• Incentives are related to organizational and social factors.• Seamless integration of new features.
• Semantic tools– Game mechanics.– Paid crowdsourcing (integrated).
• Using results of casual games.
http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/
Case studies
• Methods applied– Mechanism design.– Participatory design.– Games with a purpose.– Crowdsourcing via MTurk.
• Semantic contentauthoring scenarios– Extending and populating
an ontology.– Aligning two ontologies.– Annotation of text, media
and Web APIs.
Mechanism design in practice
• Identify a set of games that represents your situation.• See recommendations in the literature.
• Translate what economists do into concrete scenarios.• Assure that the economists’ proposals fit to the concrete situation.
• Run user and field experiments. Results influence HCI, social and data management aspects.
8/26/2011 www.insemtives.eu 17
Factors affecting mechanism design
8/26/2011 www.insemtives.eu 18
Goal Tasks Social Structure
Nature of good being produced
Communication level (about the
goal of the tasks)
High
Variety of
High
Hierarchy neutral
Private goodMedium Medium
Low Low
Participation level (in the definition
of the goal)
High
Specificity of
High
Public goodMedium Medium
Low Low
Clarity levelHigh Identification
withHigh
HierarchicalCommon resourceLow
Low Required skillsHighly specific
Club goodTrivial/Common
More at http://www.insemtives.eu/deliverables/INSEMTIVES_D1.3.1.pdf andhttp://www.insemtives.eu/deliverables/INSEMTIVES_D1.3.1.pdf
Phase 3: OKenterprise annotation tool
4/14/11 www.insemtives.eu 19
Mechanism design for Telefonica
• Interplay of two alternative games– Principal agent game
• The management wants employees to do a certain action but does not have tools to check whether employees perform their best effort.
• Various mechanisms can be used to align employees’ and employers’interests
– Piece rate wages (labour intensive tasks)– Performance measurement (all levels of tasks)– Tournaments (internal labour market)
– Public goods• Semantic content creation is non-rival and non-excludable• The problem of free riding
• Additional problem: what is the optimal time and effort for employees to dedicate to annotation
4/14/11 www.insemtives.eu 20
Mechanism design for Telefonica (ii)
• Principal agent game– Pay-per-performance
• Points assigned for each contribution
– Quality of performance measurement
• Rate user contributions• Assign quality reviewers
– Tournament• Visibility of contributions by
single users• Search for an expert based on
contributions• Relative standing compared to
other users
• Public goods game– To let users know that their
contribution was valuable– The portal should be useful
• Possibility to search experts, documents, etc.
• Possibility to form groups of users and share contributions
– The portal should be easy to use
• Experiments– Pay-per-tag vs winner-takes-
it-all for annotation.
4/14/11 www.insemtives.eu 21
Tasks in knowledge engineering• Definition of vocabulary• Conceptualization
– Based on competency questions– Identifying instances, classes, attributes,
relationships
• Documentation– Labeling and definitions.– Localization
• Evaluation and quality assurance– Matching conceptualization to documentation
• Alignment• Validating the results of automatic methods
www.insemtives.eu 22
23
http://www.ontogame.orghttp://apps.facebook.com/ontogame
OntoGame API• API that provides several methods that are
shared by the OntoGame games, such as: – Different agreement types (e.g. selection
agreement).– Input matching (e.g. , majority).– Game modes (multi-player, single player). – Player reliability evaluation. – Player matching (e.g., finding the optimal
partner to play).– Resource (i.e., data needed for games)
management.– Creating semantic content.
• http://insemtives.svn.sourceforge.net/viewvc/insemtives/generic-gaming-toolkit
www.insemtives.eu 248/26/2011
OntoGame games
8/26/2011 www.insemtives.eu 25
www.insemtives.eu
SEAFish – Annotating images
8/26/2011 26
Lessons learned• Approach is feasible for mainstream domains, where a
(large-enough) knowledge corpus is available.• Advertisement is important.• Game design vs useful content.
– Reusing well-kwown game paradigms.– Reusing game outcomes and integration in existing workflows
and tools.
• But, the approach is per design less applicable because– Knowledge-intensive tasks that are not easily nestable. – Repetitive tasks players‘ retention?
• Cost-benefit analysis.
Using Mechanical Turk forsemantic content authoring
• Many design decisions similar to GWAPs.– But clear incentives structures.– How to reliably compare games and MTurk results?
• Automatic generation of HITs depending on thetypes of tasks and inputs.
• Integration in productive environments.– Protégé plug-in for managing and using crowdsourcing
results.
