icml- crowdml - ioannis (yanni) antonellis, senior director, data science at upwork

© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.

Machine Learning at Upwork

Yanni AntonellisSenior Director, Data [email protected]://www.linkedin.com/in/ioannisantonellis

ICML 2015 Workshop on Crowdsourcing and Machine Learning

mailto:[email protected]

https://www.linkedin.com/in/ioannisantonellis


Outline

• Machine Learning at Upwork

• Upwork Freelancer Search

• Upwork Job Search (Alya Abbott)

2

© 2015 Upwork Inc. Proprietary and confidential. Do not distribute. 3

The Earth-sized talent pool


Competitive Time-consuming

Expensive Inflexible

“Traditional” work models, remnants of the Industrial Age, don’t make sense today

4

There are 53 Million freelancers in the U.S. today

Source: Staffing Industry Analysts

By 2020, nearly half of the U.S. workforce will be freelancing

$47B online staffing

$45B

2013 2020

Source: Freelancers Union

$47B


Commerce has moved online and now services, including work, are too

6

workretail lodgingtransportation


Online Platforms Enabling Steady Growth in Product Commerce & Enabling Rapid Growth in Service Commerce

7

Internet Trends Report by KPCB & Mary Meeker

http://www.kpcb.com/internet-trends


The Internet is becoming the road into work

8


Future teams will look like movie crews

9


Companies that aren’t distributed can’t possibly

say they hire the best people…the world is a big place.Tom Preston-Werner, Founder of GitHub


4M+businesses

seeking 2,500+ skills

10M+freelancers

from 180+ countries

$1B+worth of work

annually

Upwork today

Our marketplace connects clients with talent in a snap

2.4MJobs

posted annually

22Candidates

per job

24 hrs25% of jobs

are filled


Clients of all sorts hire and manage online

Startups Mid-sized Large

5+ freelancers

30+ freelancers

300+ freelancers

13


Freelancers find clients and work online

14

Small agencies & independent professionals

Moonlighters Studentsand retirees

Anna · Designer

UK

Stan · Data Scientist

Florida

Esteban · DeveloperArgentina

In a growing number of categories

Earnings by Category

Data

We have the biggest closed-loop online dataset of jobs and job seekers in labor history

Resumes (~10M)Job Posts (~10M)Job Applications (~100M)Messages (~100M)Hiring decisions (~10M)

Contract progress (~1B)Feedback(~10M)Web site activity (~10B)Money transactions (~100M)

Match: How clients find freelancers

1. Freelancer search

2. Similar freelancers

3. Freelancerrecommendation

s

Match: How freelancers find interesting jobs

1. Job feed/recommendations

2. Job search

3. Similar jobs

Match: How clients pick the best applicants

1. Sort by Best Match

2. Filter spam

3. Guaranteed applicants

Match: What makes it hard

Personalization

Two-sided marketconstraints

“gangnam style video” “iOS developer”

Each contractor has ~170 work hours

per monthin 7 months


Upwork Freelancer Search

21

1. POST – APPLYContactors apply to posted jobs, clients evaluate the applicants and decide which one to hire

2. SEARCH – INVITEClients search for contractors and invite them to work on a posted job

Two ways for contractors and clients to find each other...

Upwork Freelancer Search

Upwork Freelancer Recommendations


Client Funnel & Search/Recommendations

SignupJob Posting

InvitationsAccepted Invitations

HiresGood Outcomes

$

26



Visitor site search

User site Search & Recommendations

SignupJob Posting


HiresGood Outcomes

$

27



Visitor site search


Search before a

job posting

Search after a

job posting

Freelancer recs after a job posting

Freelancer recs after

an invitation

Applicant Ranking

SignupJob Posting


HiresGood Outcomes

$

28



Visitor site search


Search before a

job posting

Search after a

job posting

Freelancer recs after a job posting

Freelancer recs after

an invitation

Applicant Ranking

SignupJob Posting


HiresGood Outcomes

$

29

Ranking


Client Funnel and Search

• Different flavors of Ranking are needed per Search Product since:

– (a) the objective we are optimizing for and

– (b) the available features vary.

• For example, when a client searches for freelancers before he has posted any job we probably want to optimize the results for encouraging the client to post a job or contact a freelancer.

• However, when a client is searching *after* posting a job, we have the job context information and we probably want to optimize the search results for accepted invitations or hires.

30


Optimization Objectives

Data Product Optimization Objective

Visitor search Signups, FJPs

Search before a job posting Job postings, Invitations

Search after a job posting Accepted Invitations, Hires, Good Outcomes

Freelancer recommendations after a job posting

Accepted Invitations, Hires, Good Outcomes

Freelancer recommendations after an invitation

Accepted Invitations, Hires, Good Outcomes

Applicant Ranking Hires, Good Outcomes

SignupJob Posting


HiresGood Outcomes

$

31


Optimization Objectives

Data Product Optimization Objective

Visitor search Signups, FJPs

Search before a job posting Job postings, Invitations

Search after a job posting Accepted Invitations, Hires, Good Outcomes

Freelancer recommendations after a job posting

Accepted Invitations, Hires

Freelancer recommendations after an invitation

Accepted Invitations, Hires

Applicant Ranking Hires, Good Outcomes

SignupJob Posting


HiresGood Outcomes

$

32


Search after posting a job - Funnel

33

Impressions Clicks InvitationsAccepted Invitations

HiresGood Outcomes


Search after posting a job


HiresGood Outcomes

Relevance Availability Eagerness Quality

34


Relevance

• Freelancer summary

• Job summary

• Freelancer - Job fit

• Freelancer - Client fit

35


Availability

Sample Features

● Recent hours worked

● Recent invitations received

● Recent applications placed

● Recent site location use:

o Contractor profile

o Job search

o Message center

● Contractor-provided availability

● Historical wages & current quote

36


HiresGood Outcomes


Eagerness

AUC of 0.792 - Sample features:

● Job applications involving the freelancer:

o Job apps sent/invites received in previous {1, 2, 4, 7, 14, 30} days

o Job apps/invites filled in previous {1, 2, 4, 7, 14, 30} days

o Job apps/invites rejected in previous {1, …, 30} days

o Job apps/invites cancelled in previous {1, …, 30} days

● # of Eventlog entries at {message_center, job_search,

contractor_profile_page} locations in previous {1, …, 30} days

● Hours worked in previous {1, …, 30} days

● Profile details:

o Num fixed/hourly jobs worked

o Typical wage; rate quote

o Self-reported availability

o Num. active assignments and candidacies

o Num. jobs worked in each category group

37


HiresGood Outcomes


Quality

AUC of 0.61 - Sample features:

● Average number of Good outcomes in the past● Average number Bad outcomes in the past● History of Freelancers in terms of receiving/giving bad or good private

feedbacks● Profile exams taken (e.g. PHP test, Book keeping test, Spanish Spelling Skills

test )● Freelancers Skills (e.g. Editing, Translation, Proofreading, . . . )

38


HiresGood Outcomes


Search after posting a job


HiresGood Outcomes

Relevance Availability Eagerness Quality

Visibility Rotation

39


Visibility Rotation


Summary

• Many flavors of freelancer search based on:

– optimization objective, context

– data/features available

• Key building blocks

– Relevance

– Availability

– Eagerness

– Quality

– Visibility Rotation42


Hiring in 3 minutes is … magic


ThanksYanni Antonellis, Senior Director, Data Science

[email protected]

icml- crowdml - ioannis (yanni) antonellis, senior director, data science at upwork

Data & Analytics

similar freelancers

freelancers union

b online staffing

job seekers

b worth of work

job posts

job applications

job feedrecommendations