insemtives tutorial iswc2011 - session3

Crowdsourcing the annotation of dynamic Web content at seekda

www.insemtives.eu 1

Elena Simperl, University of Innsbruck, AT Markus Rohde, University of Siegen, DE

ISWC 2010

Overview

• Context • Prototyping • Participatory Design • User Challenges • Conclusions

www.insemtives.eu 2

Context

• Web services portal – Crawls for and indexes Web Services on the Web – Currently more than 28,500 indexed and monitored

• Problems – Services are not annotated or described – Limited search results and possibilities – Web APIs need to be confirmed by users

• Goal – Obtain more annotations by involving users in the annotation

process – Validate existing annotations, if any – „Catch them & keep them“

www.insemtives.eu 3

Design decisions • Different annotation methods exist

– Keywords/tags – Categories – Natural language descriptions – Lightweight/fully-fledged semantic web service descriptions (e.g.

WSMO/Light, OWL-S, etc.) – Avoid complicated and demanding annotations (limit to tags, categories and

NL descriptions) • Use lightweight RDF ontologies in the background (e.g. to ease the

search) • SWS annotations might be integrated in the future

– Most users are not familiar with SWS – Difficult to integrate within the search (diverse frameworks and variants) – May hamper performance & usability

www.insemtives.eu 4

Prototype Creation

www.insemtives.eu 5

Design Recommendations

www.insemtives.eu 6

Participatory Design

• Involve (end) users in prototyping • Users = experts of use/practice • Needs assessment -> Requirements Analysis • Exploit users‘ expertise and creativity in design

processes • Integrate evaluation in design processes • Repeated prototyping cycles

www.insemtives.eu 7

Online Participatory Design

• Seekda‘s users are – anonymous – distributed worldwide

• Online communication via website • Creating opportunities for online participation

– Establish appropriate OPD process design – Develop adequate OPD tool (= dashboard)

www.insemtives.eu 8

Stakeholders‘ Benefits

www.insemtives.eu 9

Users‘ Benefit

• Design follows users‘

needs

• Implement own ideas

• Insights in technology

and development

Seekda‘s Benefit

• Getting direct input from

users/customers

• Focussing on central

user requirements

• Getting to know

users/customers

OPD Stakeholders/ Process Roles

• Project owner – Initiation, Management, Coordination – Facilitation

• Research/ Observer – Expert as neutral consultant

• Technical committee – Developers/Designers and Users – Process Decisions

• User committee

www.insemtives.eu 10

OPD Workshop - Procedure • In General:

1. Technical committee chooses number of features out of forum discussions

2. Features open for user voting 3. Feature selection, implementation 4. Collect user feedback

• Duration per cycle: 6 weeks


OPD Workshop – Cycles •1st Cycle

• Beginning November: Start of workshop (dashboard, technical committee, introduction)

• 2 weeks later: Identification of 5 most important features/wishes • 1 week later: Selection of 2 most popular features • Beginning of Dec: Short tasks for users • Mid December: End of cycle, feedback analysis • OPD Dashboard Improvements

•2nd Cycle • Execution dates: January – March • Goals for this cycle

• Increase motivation • Increase activity of participants • Focus more on usability/design and incentives

• Changes • Tasks first • Split into smaller parts, sequentially • Explained through screencasts • Example: go to the portal, search for xyz, identify redundant elements, most

important, … • OPD Dashboard Improvements


PD of the Dashboard

On-site PD workshop: Requirements for the PD dashboard


OPD Description


Process description for participants + Video Instruction

OPD Announcement

13 www.insemtives.eu 15

OPD Introduction

13 www.insemtives.eu 16

OPD Dashboard (2nd Cycle)


Improvements: awareness feature and weekly tasks for participants

Feature Selection and Voting


OPD Workshop - Results • Numbers

~ 250 votes ~ 160 forum posts 15-20 active users

• User Background

• Web Services experts • Developers • Random visitors

• Feedback/Implementation • 18 suggested features • 6 concrete features implemented (ongoing) • Several implemented usability/design improvements

• Conclusions & Next Steps (ongoing) • Introduce challenge procedures • Ask specifically about guided processes (wizards) • Integrate OPD workshop directly from the platform


Evalution (I) • Six interviews (~60 min) with participants

– Experiences – General evaluation – Critique, improvements

• Limitations of written communication -> Multi-media • Performance problems • Positive: Video instruction • Improvement: Awareness features/ notifications


Evaluation (II) • Central features and Usability have been

improved • High quality feedback from users • Improved planning of features/

implementation based on early discussion with users

• (Perceived) assistance/ support for developers/ designers

• „Yeah, I think it succeeded. We got a lot of contribution from people […] and it showed this kind of workshop can work. This kind of methods.“


Challenge 1: Amazon Mechanical Turk

Goal: Initial annotations for new and undescribed APIs Tasks available

• Confirm document is related to a Web API (yes/no) • Provide/improve description • Provide and confirm (bootstrapped) tags/concepts • Provide and confirm (bootstrapped) service categories • Rate document quality

Parameters • Qualification test • Min. approval rate per worker • Approx. Time needed per task • Reward


Challenge 1: MTurk

Simple Annotation Wizard Phase 1

1. Setup • Initial set of 70 Web APIs • No qualification needed • Approx. Time needed: 15 minutes • Reward: 0,10$ • Description + Screencast (Walkthrough)

2. Manual evaluation (seekda) • Main focus on description and Yes/No question • Determine whether qualification is needed for workers • Determine whether wizard is understandable (usability) • Determine whether review tasks are needed


Challenge 1: Mturk Phase 1 Results

Total: 70 API documents, 23 distinct workers Initial Question (Document about Web API, Yes/No)

• 49 documents correctly annotated (70%) • 21 as Yes, this document is about a Web API • 28 as No, this document is not related to a Web API

Description, Category, Tags

• ~ 15 submissions including all answers (description, category, tags) • 4 very good and extensive submissions • 8 complete and usable submissions

Phase 1, Problems

• Spam (10% - 15%) • Only few added category and descriptions • Most workers did not add tags


Mturk: Phase 2 Changes

• Completion time decreased to 10min • Reward increased to 20c • Key questions are mandatory (description, tags,

category) • More strict evaluation criteria

– e.g.: at least, 1 category, 2 tags and a meaningful description have to be provided.

• Submitted a batch of 100

MTurk: Phase 2 Results • 27 users (only 1 from the previous batch!) • Completion Times

– Min: 10 sec – Max: 9 min

• 10 wrong classifications – 5 of them are web pages with high quality annotations

• For correct classifications: – Mostly annotated with 2 tags – Top level category identification accurate in most cases – Mostly meaningful descriptions – Over 80% are accurate/satisfying

MTurk: Phase 2 Results •Large number of confirmed Web APIs

–Feed back to crawler/analysis framework –Improving the initial index quality

• Large number of categorised/tagged Services/APIs –Feed back to bootstrapping Service –Improved search/navigation

• Detailed service descriptions for many Services at once

–Improved user experience –Improved user experience/satisfaction –Attract/motivate new visitors to participate

Challenge 2: Mashups • Overall Goal

– Create and add mashup(s) using services / Web APIs listed on the seekda portal – Annotate used Services and APIs

• Timeline – Duration: 4 weeks

• Introduction of Task through: – Step by step guidelines – Set of rules – Example walkthrough

• Reward – Gadget (Samsung Galaxy S)


Challenge 3: Long-Term Competition

• Provide annotations – become a top contributor • Collect Points

• Changes and/or improvements to annotations • New annotations • Weighting according to annotation type

• Rank contributors • Reputation is main award • Allow users to donate money for good annotations

• Donated money will be awarded to the top annotators • The more and better annotations provided… • …the higher the reputation • …the higher the financial incentive


Conclusion • Devising motivation methods for annotating Web

services is challenging • Different possibilities were/are being explored through

challenges – Mechanical Turk – Mashups Challenge – Long-Term Competition

• Users were closely kept in the development loop through OPD – Ensures that implemented features are usable – Keeps users engaged in a “community”-like way


Questions & Annotations

Thank You!


insemtives tutorial iswc2011 - session3

Technology