reactive crowdsourcing

REACTIVE CROWDSOURCING

Alessandro Bozzonab

Marco Brambillaa

Stefano Ceria

Andrea Mauria

aPolitecnico di Milano

Dipartimento di Elettronica, Informazione e BioIngegneria

bDelft University Of Technology

Department of Software And Computer Technology

2

Crowd Control is tough… • There are several aspects that makes crowd

engineering complicated • Task design, planning, assignment• Workers discovery, assessment, engagement

Wednesday, May 15 Reactive Crowdsourcing

http://xkcd.com/1060/

3

Crowd Control is tough…


• There are several aspects that makes crowd engineering complicated • Task design, planning, assignment• Workers discovery, assessment, engagement

• Goal: taming the crowd• Cost• Time• Quality

4

Crowd Control is tough…


• There are several aspects that makes crowd engineering complicated • Task design, planning, assignment• Workers discovery, assessment, engagement

• Goal: taming the crowd• Cost• Time• Quality• Motivation!

• Need for higher level abstractions and tools• CONTROL as first-class citizen

5

Reactive Crowdsourcing• A conceptual framework for modeling crowdsourcing

computations and control requirements• Task Design• Reactive Control Design

• Active Rule programming framework• Declarative rule language

• A reactive execution environment for requirement enforcement and reactive execution• Based on the CrowdSearcher approach


6

Why Active Rules?• Crowdsourcing control typically focuses on task data

• Execution results, agreement on truth value, workers performance

• An active rule approach can provide• Ease of Use: control is easily expressible

• Simple control data structures • Familiar formalism

• Power: support to arbitrarily complex controls• Extensibility mechanisms

• Automation: most active rules can be system-generated• Well-defined semantic

• Flexibility: simple control variants have localized impact on the rules set• Control isolation


7

The CrowdSearcher Approach• Human-Enhanced data management with social networks

and Q&A systems as crowdsourcing platforms

• Example: search task (WWW2012)


Human Interaction Management

Social Networks

HumanComputation

PlatformsQ&A

Data Management System

Human Access Interface

Remote Data Access

Local Data Access

Task

Human-EnhancedData

Query

Results

8

• A simple abstract model• A task receives a list of input objects • Performers execute one or more operations upon them• The task produces a list of crowd-manipulated objects

• A simple task design and deployment process, based on specific data structures• created using model-driven transformations• driven by the task specification

The Design Process


More in the paper

I O

9



The Design Process

Task Specification Task Planning Control Specification


• Task Spec: task operations, objects, and performers Dimension Tables

10



The Design Process



• Task Spec: task operations, objects, and performers Dimension Tables• Task Planning: work distribution Execution Table for task monitoring

11



The Design Process



• Task Spec: task operations, objects, and performers Dimension Tables• Task Planning: work distribution Execution Table for task monitoring• Control Specification: task control policies Control Mart

12

Task Specification_1/3


• Operation Types: Choice, Like, Score, Tag, Classify, Order, …• Operation Parameters: e.g. classification classes

Task

tIDopType

categories


Task Configuration

t1Classify

Rep/Dem

13



Politician

classifiedParty

lastNamephoto

oID

• Input Objects Schema: typed attributes• Output Attributes (according to task type)

Task

tIDopType

categories


Task ConfigurationObject

Specification

o1 Obama http://…. ?????

14



Politician

classifiedParty

lastNamephoto

oID

Task ConfigurationObject

SpecificationPerformer

Specification

• Execution platform(s)• Qualifications, etc.

Task

tIDopType

categories PerformernamepID

platform


p1 Alessandro Facebook

15

Task Planning_1/2

• Organize the task in MicroTasks, and allocate input objects• μTaskObjectExecution Designed for execution monitoring

• Track performers response

classifiedPartyplatform

μTaskObjectExecution

μtIDstartTs

endTsoIDpID


Politician

classifiedParty

lastNamephoto

oID

Task

tIDopType


platform

Splitting


mt1 O1 … …

… …

Facebook

16

Task Planning_2/2

• Assign performers to MicroTasks on platforms• Pull: dynamic assignment (First come - First served / Choice of the

performer)• Push: static assignment (Performers’ priority / Performer matching)



μtIDstartTs

endTsoIDpID


Politician

classifiedParty

lastNamephoto

oID

Task

tIDopType


platform

Splitting Assignment


mt1 O1 P1 Republican

00:00:01 00:00:10 Facebook

17

Control Specification_1/4



• Status Variable: tracking task and performers status



μtIDstartTs

endTsoIDpID

Politician

classifiedParty

lastNamephoto

oID

PerformernamepID

statusplatformTask

tIDopType

categoriesstatus Trusted/SpammerCreated/Planned/Closed

18




• Object : tracking objects status



μtIDstartTs

endTsoIDpID

Politician

classifiedParty

lastNamephoto

oID

PerformernamepID

statusplatformTask

tIDopType

categoriesstatus

ObjectControl #dem

oID#eval

#rep

#curAnswer

19




• Object : tracking object responses• Performer: tracking performer behavior (e.g. spammers)

PerformerControl #right

pID#eval

#wrong



μtIDstartTs

endTsoIDpID

Politician

classifiedParty

lastNamephoto

oID

PerformernamepID

statusplatformTask

tIDopType

categoriesstatus

ObjectControl #dem

oID#eval

#rep

#curAnswer

20


• Object : tracking object responses• Performer: tracking performer behavior (e.g. spammers)• Task: tracking task status: closing @completion, re-plan


TaskControl#compObj

tID#compExec

PerformerControl #right

pID#eval

#wrong



μtIDstartTs

endTsoIDpID

Politician

classifiedParty

lastNamephoto

oID

PerformernamepID

statusplatformTask

tIDopType

categoriesstatus

ObjectControl #dem

oID#eval

#rep

#curAnswer


21

Active Rules Language• Active rules are expressed on the previous data structures• Event-Condition-Action paradigm


22


• Events: data updates / timer• ROW-level granularity

• OLD before state of a row• NEW after state of a row


e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]

23




• Condition: a predicate that must be satisfied (e.g. conditions on control mart attributes)



c: NEW.ClassifiedParty == ’Republican’

24




• Condition: a predicate that must be satisfied (e.g. conditions on control mart attributes)

• Actions: updates on data structures (e.g. change attribute value, create new instances), special functions (e.g. replan)



c: NEW.ClassifiedParty == ’Republican’

a: SET ObjectControl[oID == NEW.oID].#Eval+= 1

More in the paper

25Wednesday, May 15 Reactive Crowdsourcing

e: UPDATE FOR ObjectControl

c: (NEW.Rep== 2) or (NEW.Dem == 2)

a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,

SET TaskControl[tID==NEW.tID].compObj += 1

Rule Example

TaskControl#compObj

tID PerformerControl


Politician classifiedPartyoID

PerformerTask

ObjectControl #dem

oID

#rep

#evaltIDEvent






Rule Example

TaskControl#compObj




PerformerTask

ObjectControl #dem

oID

#rep

#evaltID

Condition






Rule Example

TaskControl#compObj




PerformerTask

ObjectControl #dem

oID

#rep

#evaltID

Action

29

Rule Programming Best Practice• We define three classes of rules



PerformerControl

ObjectControl

TaskControl

Politician Performer Task

30

Rule Programming Best Practice


• We define three classes of rules• Control rules: modifying the control tables;


PerformerControl

ObjectControl

TaskControl


31



• We define three classes of rules• Control rules: modifying the control tables; • Result rules: modifying the dimension tables (object, performer, task);


PerformerControl

ObjectControl

TaskControl


32



• Top-to-bottom, left-to-right, evaluation• Guaranteed termination

• We define three classes of rules• Control rules: modifying the control tables; • Result rules: modifying the dimension tables (object, performer, task);


PerformerControl

ObjectControl

TaskControl


33

Rule Programming Best Practice• We define three classes of rules

• Control rules: modifying the control tables; • Result rules: modifying the dimension tables (object, performer, task); • Execution rules: modifying the execution table, either directly or through re-planning



PerformerControl

ObjectControl

TaskControl


• Termination must be proven (Rule precedence graph has cycles)

34

Experimental Evaluation

• GOAL: demonstrate the flexibility and expressive power of reactive crowdsourcing

• 3 experiments, focused on Italian politicians• Parties: Human Computation affiliation classification• Law: Game With a Purpose guess the convicted politician• Order: Pure Game hot or not

• 1 week (November 2012)• 284 distinct performers

• Recruited through public mailing lists and social networks announcements

• 3500 Micro Tasks


35

Politician Affiliation• Given the picture and name of a politician, specify his/her political

affiliation• No time limit• Performers are encouraged to look up online

• 2 set of rules• Majority Evaluation• Spammer Detection


36

Results – Majority Evaluation_1/3


30 object; object redundancy = 9;

Final object classification as simple majority after 7 evaluations

37

Results - Majority Evaluation_2/3


Final object classification as total majority after 3 evaluations

Otherwise, re-plan of 4 additional evaluations. Then simple majority at 7

38

Results - Majority Evaluation_3/3


Final object classification as total majority after 3 evaluations

Otherwise, simple majority at 5 or at 7 (with replan)

39

Results – Spammer Detection_1/2


New rule for spammer detection without ground truth

Performer correctness on final majority. Spammer if > 50% wrong classifications

40

A Short CrowdSearcher Demo


41

Summary• Results

• An integrated framework for crowdsourcing task design and control• Well-structured control rules with some guarantees of termination• Support for cross-platform crowd interoperability• A working prototype crowdsearcher.search-computing.org

• Forthcoming• Exploitation of interoperability• Expertise finding• Dynamic planning• Integration with other social-networks and human computation

platforms


42

QUESTIONS?


reactive crowdsourcing

Technology

simple task design

task operations

task monitoring

reactive crowdsourcing

crowdsourcing control

search task

crowd control

task data execution