crowd work cv: recognition for micro work
TRANSCRIPT
Institute for Web Science and Technologies · Univ ersity of Koblenz-Landau, Germany
Crowd Work CV:Recognition for Micro Work
Cristina Sarasua and Matthias Thimm
SoHuman 2014, Barcelona
Crowd Work CV: Recognition for Micro Work 3Cristina Sarasua
“Better theoretical models, markets or automatic computational matching processes could drastically reduce development costs and
address search friction, an important issue in labor economics“
The Future of Crowd Work (Kittur et al., 2013)
Task assignment Career ladders Transparency
Crowd Work CV: Recognition for Micro Work 4Cristina Sarasua
Motivational Scenario
Is Alice a good candidate for this micotask?
Spanish qualification
� Same job
� Tweet sentiment analysis
Alice
Alice´s previous work
Crowd Work CV: Recognition for Micro Work 5Cristina Sarasua
Motivational Scenario
Is Alice a good candidate for this micotask?
Spanish qualification
� Same job
� Tweet sentiment analysis
Alice
Alice´s previous work
Crowd Work CV: Recognition for Micro Work 7Cristina Sarasua
Curriculum Vitae in Microtask Crowdsourcing
• RDF data model to describe a microtaskcrowdsourcing agent's• interests
• obtained qualifications
• work history
• It enables the representation of crowdsourcingagents’ identities and promotes their workexperience across different microtask marketplaces
Crowd Work CV: Recognition for Micro Work 9Cristina Sarasua
Requirements
1. Domain independence
2. Marketplace independence
3. Semantic and syntactic interoperability
4. Extensibility
5. Compatibility with traditional CV information
Crowd Work CV: Recognition for Micro Work 13Cristina Sarasua
Why Semantic Web technologies?
� agreed upon vocabulary� integration of Crowd Work
CV data (e.g. with marketplaces,other applications, Web data sets)
� graph of requesters,microtasks and contributors
� extensible data model thatenables knowledgeinference
Picture: Linking Open Data cloud diagram 2014, by Max Schmachtenberg, Christian Bizer, AnjaJentzsch and Richard Cyganiak. http://lod-cloud.net/"
Crowd Work CV ontology (OWL):https://github.com/criscod/CrowdWorkCV
Crowd Work CV: Recognition for Micro Work 14Cristina Sarasua
Exampleex:rex a cwcv:RequesterExperience;
cwcv:byUserAccount ex:acc1;
cwcv:basedOn ex:mm1;
ex:cv1 cwcv:hasRequesterExperience ex:rex;
e:mm1 a cwcv:MasterMicrotask;
cwcv:publishedAt ex:ClixSense;
cwcv:publishedAt ex:GetPaid;
cwcv:purpose microtaskpurposes:SentimentAnalysis;
ex:mm2 a cwcv:MasterMicrotask;
cwcv:hasQualification ex:q1;
cwcv:publishedAt ex:Neobux;
Data described using theCrowd Work CV ontology
Crowd Work CV: Recognition for Micro Work 15Cristina Sarasua
Example (cont.)ex:mex1 a cwcv:WorkerExperience;
cwcv:basedOn ex:m1;
cwcv:basedOn ex:mi;
cwcv:byUserAccount ex:acc2;
cwcv:accuracy "0.9";
ex:m1 a cwcv:Microtask;
cwcv:hasMasterMicrotask ex:mm1;
ex:q1 a cwcv:Qualification;
cwcv:name "Spanish A1";
cwcv:isRelatedToComptenece disco:Capability1;
ex:cv2 cwcv:hasWorkerExperience ex:mex1;
cwcv:obtainedQualification ex:q1;
Crowd Work CV: Recognition for Micro Work 16Cristina Sarasua
Verification
� Verified correctness with OOPS! Pitfall Scanner http://oeg-lia3.dia.fi.upm.es/oops/index-content.jsp
� Requirements fulfilment1. Domain attached separately (e.g. SKOS vocabularies) � Domain
independence
2. Abstract concepts (e.g. evaluation) � Marketplace independence
3. Ontology defined separately with OWL � Semantic and syntactic interoperability
4. Subclasses, subproperties � Extensibility
5. Work experience, educational achievements, related skills and competences � Compatibility with traditional CV information
Crowd Work CV: Recognition for Micro Work 17Cristina Sarasua
Differences from Related Work
# matching tasks and workers No shareable and reusable description of worker expertise thatcan be used across-platforms
# vocabularies in crowdsourcingDescribes only tasks, users, rewards and capabilities primarilyfor routing. We describe work experience, microtasks,marketplaces, qualifications, requesters’ info; CV summaries
# CV / HCOMP-PROV vocabulariesLacking crowd work-specific nuances
Khazankin and colleagues, Goeland colleagues, Difallah and colleagues
Ul Hassan and colleagues
Bojars, Celino
Crowd Work CV: Recognition for Micro Work 18Cristina Sarasua
Crowd Work CV: FAQs
• Is a Crowd Work CV a userprofile? It is more than that! Workhistory plays a key role
• Who should own and share theCrowd Work CV data? The agentrepresented in the CV (e.g.crowd worker)
• How to avoid spam? Withcertified information
Crowd Work CV: Recognition for Micro Work 22Cristina Sarasua
Conclusions
� Crowd Work CV is an approach for modelling andsharing knowledge about crowd work experience ,which enriches the way reputation and credentials aremanaged in the current crowd workplace.
� Crowd Work CV could facilitate a fruitful requester-crowdworker interaction in microtask marketplaces and weaverelations of trust .
Crowd Work CV: Recognition for Micro Work 23Cristina Sarasua
Future research directions
To what extent do crowd workers (and requesters) work (and publish work) across different marketplaces?
How to extract summarized information out of many Crowd Work CV RDF triples?
To what extent does considering cross-platform CV information while selecting candidates, improve the
accuracy of results?
Crowd Work CV: Recognition for Micro Work 24Cristina Sarasua
Possible topics for the discussion
� What kind of information would you expect from a Crowd Work CV if youwere a …
� … requester?
� … crowd worker?
� How would you like Crowd Work CVs to be used if you were a …
� … requester?
� … crowd worker?
� Which arguments would you need for adopting the Crowd Work CV approach in your marketplace, if you were a marketplace owner (e.g. MTurk)?
� What pros and cons do you identify?
� Any other comment or question is welcome!
Institute for Web Science and Technologies · Univ ersity of Koblenz-Landau, Germany
Thank you for your attention !
Cristina SarasuaInstitute for Web Science and TechnologiesUniversität [email protected]
Crowd Work CV: Recognition for Micro Work 26Cristina Sarasua
References
Kittur, A., Nickerson, J.V., Bernstein, M.S., Gerber, E.M., Aaron, S.,Zimmerman, J., Lease, M., Horton, J.J.: The future of crowd work. In: 16thACM Conference on Computer Supported Cooperative Work (CSCW 2013)(2013)
Difallah, D.E., Demartini, G., Cudré-Mauroux, P.: Pick-a-crowd: tell me whatyou like, and I’ll tell you what to do. In: Proceedings of the 22nd internationalconference on World Wide Web (WWW2013) (2013)
Gagan Goel, A.K., Singla, A.: Matching workers expertise with tasks:Incentives in heterogeneous crowdsourcing markets. In: NIPS13 Workshopon Crowdsourcing: Theory, Algorithms and Applications (2013)
Ul Hassan, U., O’Riain, S., Curry, E.: Slua: Towards semantic linking of userswith actions in crowdsourcing. In: CrowdSem (2013)
Khazankin, R., Psaier, H., Schall, D., Dustdar, S.: QoS-based taskscheduling in crowdsourcing environments. In: Proceedings of the 9thinternational conference on Service-Oriented Computing (2011)
Crowd Work CV: Recognition for Micro Work 27Cristina Sarasua
References (II)
Bojars, U., Breslin, J. ResumeRDF: Expressing Skill Information on theSemantic Web . In: Proceedings of the 1st International ExpertFinderWorkshop,Knowledge Web General Assembly 2007 (2007)
Celino, I. Human Computation VGI Provenance: Semantic Web-BasedRepresentation and Publishing. In: IEEE Transactions on Geoscience andRemote Sensing. (2013)
Sarasua, C., Thimm, M.: Microtask available, send us your cv! In:International Workshop on Crowd Work and Human Computation(CrowdWork2013) (2013)