combining test data mana 4328 dr. jeanne michalski [email protected]
TRANSCRIPT
Selection Decisions
First, how to deal with multiple predictors?
Second, how to make a final decision?
Developing a Hiring System
OK, Enough Assessing: Who Do We Hire??!!
Interpreting Test Scores
Norm-referenced scores Test scores are compared to applicants or
comparison group. Raw scores should be converted to Z scores or
percentiles Use “rank ordering”
Criterion-referenced scores Test scores indicate a degree of competency NOT compared to other applicants Typically scored as “qualified” vs. “not qualified” Use “cut-off scores”
Setting Cutoff Scores
Based on the percentage of applicants you need to hire (yield ratio). “Thorndike’s predicted yield” You need 5 warehouse clerks and expect 50 to apply.
5 / 50 = .10 (10%) means 90% of applicants rejected Cutoff Score set at 90th percentile Z score 1 = 84th percentile
Based on a minimum proficiency score Based on validation study linked to job analysis Incorporates SEM (validity and reliability)
Selection Outcomes
PREDICTION
PERFORMANCE
No Pass Pass
Regression LineCut Score
90% Percentile
Selection Outcomes
PREDICTIONPREDICTION
High Performer
Low Performer
True Positive
True Negative
Type 2 ErrorFalse
Positive
Type 1 ErrorFalse
Negative
PERFORMANCEPERFORMANCE
No Hire Hire
Selection Outcomes
PREDICTION
High Performer
Low Performer
PERFORMANCE
Unqualified Qualified
Prediction Line
Cut Score
Dealing With Multiple Predictors
“Mechanical” techniques superior to judgment
1. Combine predictors Compensatory or “test assessment approach”
2. Judge each independently Multiple Hurdles / Multiple Cutoff
3. Profile Matching
4. Hybrid selection systems
Compensatory Methods
Unit weightingP1 + P2 + P3 + P4 = Score
Rational weighting(.10) P1 + (.30) P2 + (.40) P3 + (.20) P4 = Score
RankingRankP1 + RankP2 +RankP3 + RankP4 = Score
Profile MatchingD2 = Σ (P(ideal) – P(applicant))2
Multiple Regression Approach Predicted Job perf = a + b1x1 + b2x2 + b3x3
x = predictors; b = optimal weight Issues:
Compensatory: assumes high scores on one predictor compensate for low scores on another
Assumes linear relationship between predictor scores and job performance (i.e., “more is better”)
Multiple Cutoff Approach Sets minimum scores on each predictor Issues
Assumes non-linear relationship between predictors and job performance
Assumes predictors are non-compensatory How do you set the cutoff scores?
Multiple Cutoff Approach Sets minimum scores on each predictor Issues
Assumes non-linear relationship between predictors and job performance
Assumes predictors are non-compensatory How do you set the cutoff scores? If applicant fails first cutoff, why continue?
Test 1 Test 2 Interview Background
FinalistDecision
Reject
Multiple Hurdle Model
Fail FailFail Fail
Pass PassPass Pass
Profile Matching Approach
Emphasizes “ideal” level of KSA e.g., too little attention to detail may produce sloppy
work; too much may represent compulsiveness Issues
Non-compensatory Small errors in profile can add up to big mistake in
overall score Little evidence that it works better
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Detail Experience C. Service Sales Apt
Profile Match Example
Ideal
Profile Match Example
0
1
2
3
4
5
6
Detail Experience C. Service Sales Apt
Ideal
J ohn
Sam
Sue
Making Finalist Decisions
Top-Down Strategy Maximizes efficiency, but may need to look at adverse
impact issues Banding Strategy
Creates “bands” of scores that are statistically equivalent (based on reliability)
Then hire from within bands either randomly or based on other factors (inc. diversity)
Banding
Grouping like test scores together Function of test reliability
Standard Error of Measure Band of + or – 2 SEM 95% Confidence interval
If the top score on a test is 95 and SEM is 2, then scores between 95 and 91 should be banded together.
Applicant Total Scores9493898887878681818079797872706967
Information Overload!!
Leads to: Reverting to gut instincts Mental Gymnastics
Combined Selection Model
Selection Stage
Selection Test
Decision
Model
Applicants Candidates
Application Blank Minimum Qualification
Hurdle
Candidates Finalists
Four Ability Tests
Work Sample
Rational Weighting
Hurdle
Finalists Offers
Structured Interview Unit Weighting
Rank Order
Offers
Hires
Drug Screen
Final Interview
Hurdle
Alternative Approach
Rate each attribute on each tool Desirable Acceptable Unacceptable
Develop a composite rating for each attribute Combining scores from multiple assessors Combining scores across different tools A “judgmental synthesis” of data
Use composite ratings to make final decisions
Who Do You Hire??
Name
Interview
Work Sample
Knowledge Test
Personality Inventory
Lee Excellent Good 90% Hire
Maria Excellent Very Good 85% Hire
Alan Good Excellent 90% Caution
Juan Marginal Good 81% Hire
Frank Excellent Poor 70% Hire
Tamika Good Good 75% Hire
Categorical Decision Approach
1. Eliminate applicants with unacceptable qualifications
2. Then hire candidates with as many desirable ratings as possible
3. Finally, hire as needed from applicants with “acceptable” ratings
Optional: “weight” attributes by importance
Sample Decision Table
Name
Customer Service
Attention to Detail
Conscient- iousness
Computer Skills
Work Knowledge
Lee Acceptable Desirable Desirable Acceptable Acceptable
Maria Desirable Desirable Acceptable Acceptable Desirable
Alan Desirable Acceptable Unacceptable Acceptable Acceptable
Juan Acceptable Acceptable Acceptable Acceptable Acceptable
Frank Desirable Desirable Desirable Unacceptable Unacceptable
Tamika Acceptable Desirable Acceptable Acceptable Acceptable
More Positions than Applicants
Name
Customer Service
Attention to Detail
Conscient-iousness
Computer Skills
Work Knowledge
Hiring Action
Lee Acceptable Desirable Desirable Acceptable Acceptable Hire
Maria Desirable Desirable Acceptable Acceptable Desirable Hire
Alan Desirable Acceptable Unacceptable Acceptable Acceptable Not Hire
Juan Acceptable Acceptable Acceptable Acceptable Acceptable Hire
Frank Desirable Desirable Desirable Unacceptable Unacceptable Not Hire
Tamika Acceptable Desirable Acceptable Acceptable Acceptable Hire
More Applicants than Positions
Name
Customer Service
Attention to Detail
Conscient-iousness
Computer Skills
Work Knowledge
Hiring Action
Lee Acceptable Desirable Desirable Acceptable Acceptable Hire 2
Maria Desirable Desirable Acceptable Acceptable Desirable Hire 1
Alan Desirable Acceptable Unacceptable Acceptable Acceptable Not Hire
Juan Acceptable Acceptable Acceptable Acceptable Acceptable Hire 4
Frank Desirable Desirable Desirable Unacceptable Unacceptable Not Hire
Tamika Acceptable Desirable Acceptable Acceptable Acceptable Hire 3
Selection
Top Down Selection (Rank) vs. Cutoff scores Is the predictor linearly related to performance? How reliable are the tests?
1. Top-down method – Rank order
2. Minimum cutoffs – Passing Scores
Final Decision
Random Selection Ranking Grouping
Role of Discretion or “Gut Feeling”