abetting the vetting: pre-hire intervention programs annual meeting handouts/atmd/na… · abetting...

Abetting the Vetting: Pre-Hire Intervention Programs

Dr. Marc Kruse Austin-Travis County Emergency Medical Services &

Austin Fire Department

Psychological & Behavioral Health Screening of Fire and EMS Applicants:

Can we Predict & Reduce “Bad” Hires? Presentation at the 9th Annual Advanced Topics in Medical Direction

at the 2017 NAEMSP Conference

Presentation Outline

� Rationale for Pre-Hire Assessment � Our Process � Evidence of Effectiveness � Changing the Culture � Suggestions for Consideration

General Assumptions � Fire and EMS jobs are highly desirable

Percent of AFD Applicants & Test Takers Hired by AFD in the Last 10 Years

Hiring Process Applied Tested Hired

-‐  2006 3185 2236 111 (3.5%/5.0%)

-‐  2008 3821 2756 58 (1.5%/2.1%)

-‐  2011 3314 2906 85 (2.6%/2.9%)

-‐  2012 4865 2670 102 (2.1%/3.8%)

-‐  2013 5297 2882 76 (1.4%/2.6%)

-‐  2015 3500 2107 105 (3.0%/5.0%)

TOTAL 24,043* 15,557* 537 (2.2%/3.5%)

General Assumptions • Fire and EMS jobs are highly desirable • Not everyone is a “Good Fit” for fire and/or EMS service • Firefighters & EMS Providers who are not a “good fit”… • Have a negative impact on a department • Pose potential safety and liability risks • Are often identified early in a career • Tendency to become “25-year problems”

• Disqualifying an applicant who is a “poor fit” for fire service or EMS during pre-hire phase has significant benefits.

Goal of Pre-Hire PBH Evaluation: Screen out applicants who are a “Poor Fit” � Values of the Department (Transportability Study Report)

� Honesty, integrity, responsibility, dependability, etc. � Personality traits and characteristics

� Self-Control, Amicability, Hostility, Tolerance, etc. � Behavioral health history & risks

� Arrests, work history, physical exercise, etc. � Stress Response

� Ability to respond to and recover from stressful experiences appropriately

Goal of Pre-Hire PBH Evaluation: Screen out applicants who are a “Poor Fit”

NOT an evaluation of: • Whether an applicant has a diagnosable psychological/

mental health condition* • How “nice” an applicant is • How much an applicant “wants” to be a firefighter or EMS

Provider and/or work for your Department

First Year Results (AFD): 17.8% Failed! • Applicant with 3 DUI Arrests (no convictions) • Applicant fired from previous Fire Job for assaulting his

Lieutenant • Applicant currently getting divorced because of a

disturbing pattern of domestic violence • Series of applicants who struggled with the truth • Military Veteran applicant with a significant service-

connected disability rating for PTSD • Former unsuccessful AFD employee*

AFD Pass/Fail Rate for Pre-Hire PBH Hiring Process Evaluated Failed PBH % Failed

-‐  2002 72 1 1.4% -‐  2003 83 1 1.2% -‐  2006 140 4 2.9% -‐  2008 95 3 3.2% -‐  2011 257 5 1.9% Pre-‐2012 647 14 2.2% -‐  2012 135 24 17.8% TOTAL 782 38 4.9%

Pre-Hire PBH Evaluation: • What are we doing? • Does it actually work? • Can we make it better? • Can I replicate our success for other departments?

Pre-Hire PBH Evaluation • Paper/Pencil Assessment (~$25) • Psychological History Questionnaire (PsyQ) • Behavioral Risk Assessment: Past “Bad” Deeds

• California Psychological Inventory (CPI) • Personality Traits & Risk Assessment • 434-Item True/False Questionnaire; Public Safety

Norms • 60-90 Minute Psychologist Interview (~$250)

Psychological History Questionnaire (PsyQ) � 14 Domains of Behavioral Health History/Risk Assessment

� Education, Employment, Military Experience, Law Enforcement Experience, Driving Record, Financial History, Legal History, Substance Use, General Information, Developmental History, Adult Relationships, Parental Responsibilities, Psychological Treatment and Evaluation History, and Job Relevant Sexual History

� Unconditional and Conditional Disqualifying Admissions (Severity, Frequency, or Recency)

� Problem Points – Percentile Rank Relative to Public Safety Applicants (since 2011)

California Psychological Inventory (CPI) � 3 Validity Profiles (“Fake Good”, “Fake Bad”, “Random”) � 7 Job Suitability Risk “Snapshots”

�  Poorly Suited Applicant, Job Performance, Integrity, Anger Management, Alcohol Use Concerns, Illegal Drug Use, Substance Abuse Proclivity

� 19 “Basic Folk” Personality Scales �  Socialization, Self-Control, Empathy, Responsibility, Tolerance, Intellectual Efficiency

� 5 Structural (Level & Type) Scales �  Level 1-7 �  Alpha, Beta, Delta, Gamma Types

� 15 Special Purpose Scales �  Narcissistic Personality, Hostility, Work Orientation, Amicability, Integrity, Self-Discipline

CPI “Fake Good” Profiles Hiring Process Evaluated % “Fake Good” % Failed

-‐  2002 72 15.3% (11) 0.0% -‐  2003 83 34.9% (29) 0.0% -‐  2006 140 30.0% (42) 2.4% (1) -‐  2008 95 20.0% (19) 0.0% -‐  2011 257 36.6% (94) 0.0% Pre-‐2012 647 30.1% (195) 0.5% (1) -‐  2012 135 38.5% (52) 11.5% (6) TOTAL 782 31.6% (247) 2.8% (7)

Is Pre-Hire PBH Effective?

q  Differentiate between applicants who failed PBH and those who passed?

q  Differentiate between firefighters and EMS providers identified as “Bad Hires” and those performing at or above standards?

q  If used properly, can it result in a reduced percentage of hired firefighters and EMS providers performing below standards?

Pre-Hire Behavioral Predictors of Unsuccessful Applicants (PsyQ)

DQ’d in 2012 (17.8%)

Rank (Civil Service Exam) -‐ ↑ PsyQ History of Work DifficulNes ü 99.9% ↑ PsyQ History of Financial DifficulNes ü 99.8% ↑ PsyQ History of Substance Use DifficulNes ü 99.9%

Pre-Hire Personality Predictors (CPI) California Psychological Inventory (CPI) Subscales:

DQ’d in 2012

↑ Poorly Suited Candidate ü 98.2% ↑ NarcissisNc Personality ü 98.2% ↑ HosNlity ü 97.5% ↓ Tolerance ü 99.9% ↓ Amicability ü 99.4% ↓ Integrity ü 99.9% ↓ Achievement Independence ü 99.8% ↓ Managerial PotenNal ü 99.5% ↓ SocializaNon ü 98.4% ↓ Intellectual Efficiency ü 96.1% ↓ Self-‐Discipline ü 98.0% ↑ Job Performance Problems ü 98.1% ↑ Substance Abuse Problems ü 99.4% ↓ Self-‐Control ü 98.2% ↓ Achievement Conformance ü 95.4%


ü  Differentiate between applicants who failed PBH and those who passed?

q  Differentiate between firefighters and EMS providers identified as “Bad Hires” and those performing at or above standards?


Who is a “Bad Hire”? • Separated from AFD • Disciplined by Professional Standards Office • Ever been suspended? (yes/no)

• Below Average Ratings from Battalion Chiefs

Battalion Chief Ratings 1.  Knowledge: Firefighter & EMT-Based tactics, policies and

procedures, territory, medical calls, etc. 2.  Skills: Firefighter & EMT-Based equipment, apparatus, fire ground

and medical procedures, physical fitness 3.  Professionalism: responsibility and dependability, attendance,

completion of other job-related tasks (CEs, station maintenance), contributions to AFD

4.  Personality: contribution to station life, ability to work well/get along with other firefighters, first-responders, and civilian employees across a variety of work-settings

5.  Overall: summary rating based on overall performance as a member of AFD

Firefighters Identified as “Bad Hires” 390 Firefighter Applicants Hired 2002-2011

2002-‐2011

No Longer with AFD (46): 11.8% -‐ 65.2% (30) Separated during academy

-‐ 15.2% (7) Separated during probaNon

-‐ 19.6% (9) Separated a^er probaNon

Disciplined by PSO (19*): 4.9%* -‐ 21.1% (4) No Longer with AFD

-‐ 46.2% (6) also rated “Below Average” by BC

Rated Below Average by BCs (63): 16.2% “Bad Hires” (110/390): 28.2%

Pre-Hire Behavioral Predictors of Unsuccessful Applicants & “Bad Hires”

DQ’d in 2012 (17.8%)

Rank -‐ Slower 1.5 Mile Run ↑ PsyQ History of Work DifficulNes ü 99.9% ↑ PsyQ History of Financial DifficulNes ü 99.8% ↑ PsyQ History of Substance Use DifficulNes ü 99.9%


DQ’d in 2012 (17.8%)

“Bad Hires” 2002-‐’11 (28.2%)

Rank -‐ -‐ Slower 1.5 Mile Run ü 99.9% ↑ PsyQ History of Work DifficulNes ü 99.9% ü 99.9% ↑ PsyQ History of Financial DifficulNes ü 99.8% ü 99.7% ↑ PsyQ History of Substance Use DifficulNes ü 99.9% -‐

California Psychological Inventory (CPI) Subscales:

DQ’d in 2012

↑ Poorly Suited Candidate ü 98.2% ↑ NarcissisNc Personality ü 98.2% ↑ HosNlity ü 97.5% ↓ Tolerance ü 99.9% ↓ Amicability ü 99.4% ↓ Integrity ü 99.9% ↓ Achievement Independence ü 99.8% ↓ Managerial PotenNal ü 99.5% ↓ SocializaNon ü 98.4% ↓ Intellectual Efficiency ü 96.1% ↓ Self-‐Discipline ü 98.0% ↑ Job Performance Problems ü 98.1% ↑ Substance Abuse Problems ü 99.4% ↓ Self-‐Control ü 98.2% ↓ Achievement Conformance ü 95.4%

Pre-Hire Personality Predictors (CPI)


DQ’d in 2012

“Bad Hires” 2002-‐’11

↑ Poorly Suited Candidate ü 98.2% ü 96.3% ↑ NarcissisNc Personality ü 98.2% ü 95.3% ↑ HosNlity ü 97.5% ü 99.4% ↓ Tolerance ü 99.9% ü 99.0% ↓ Amicability ü 99.4% ü 98.9% ↓ Integrity ü 99.9% ü 99.9% ↓ Achievement Independence ü 99.8% ü 96.3% ↓ Managerial PotenNal ü 99.5% ü 96.6% ↓ SocializaNon ü 98.4% ü 98.9% ↓ Intellectual Efficiency ü 96.1% ü 95.6% ↓ Self-‐Discipline ü 98.0% ü 94.7% ↑ Job Performance Problems ü 98.1% -‐

↑ Substance Abuse Problems ü 99.4% -‐

↓ Self-‐Control ü 98.2% -‐

↓ Achievement Conformance ü 95.4% -‐




ü  Differentiate between firefighters and EMS providers identified as “Bad Hires” and those performing at or above standards?


Firefighters Identified as “Bad Hires” 102 Firefighter Applicants Hired in 2012

2012 No Longer with AFD (6): 5.9% -‐ 33.3% Resigned prior to first day at academy

-‐ 33.3% Resigned first month of the academy

-‐ 33.3% Forced to resign/terminated

Disciplined by PSO (1): 1.0% -‐ Rated very highly by BC

Rated Below Average by BCs (9): 8.9% -‐ Assessed a^er probaNon

“Bad Hires” (16/102): 15.7%


2012 2002-‐2011 No Longer with AFD (6): 5.9% 11.8% -‐ 33.3% Resigned prior to first day at academy

-‐ 33.3% Resigned first month of the academy

-‐ 33.3% Forced to resign/terminated

Disciplined by PSO (1): 1.0% 4.9% -‐ Rated very highly by BC

Rated Below Average by BCs (9): 8.9% 16.2% -‐ Assessed a^er probaNon

“Bad Hires” (16/102): 15.7% 28.2%


DQ’d in 2012 (17.8%)

“Bad Hires” 2002-‐’11 (28.2%)

Rank -‐ -‐ Psychologist RaNngs

Slower 1.5 Mile Run ü 99.9% ↑ PsyQ History of Work DifficulNes ü 99.9% ü 99.9% ↑ PsyQ History of Financial DifficulNes ü 99.8% ü 99.7% ↑ PsyQ History of Substance Use DifficulNes ü 99.9% -‐


DQ’d in 2012 (17.8%)

“Bad Hires” 2002-‐’11 (28.2%)

“Bad Hires” 2012 (15.7%)

Rank -‐ -‐ -‐ Psychologist RaNngs ü 99.9% Slower 1.5 Mile Run ü 99.9% ü 99.4% ↑ PsyQ History of Work DifficulNes ü 99.9% ü 99.9% -‐

↑ PsyQ History of Financial DifficulNes ü 99.8% ü 99.7% -‐

↑ PsyQ History of Substance Use DifficulNes ü 99.9% -‐ -‐


DQ’d in 2012

“Bad Hires” 2002-‐’11







DQ’d in 2012

“Bad Hires” 2002-‐’11

“Bad Hires” 2012

↑ Poorly Suited Candidate ü 98.2% ü 96.3% -‐

↑ NarcissisNc Personality ü 98.2% ü 95.3% -‐

↑ HosNlity ü 97.5% ü 99.4% -‐

↓ Tolerance ü 99.9% ü 99.0% -‐

↓ Amicability ü 99.4% ü 98.9% -‐

↓ Integrity ü 99.9% ü 99.9% -‐

↓ Achievement Independence ü 99.8% ü 96.3% ü 98.4% ↓ Managerial PotenNal ü 99.5% ü 96.6% -‐

↓ SocializaNon ü 98.4% ü 98.9% -‐

↓ Intellectual Efficiency ü 96.1% ü 95.6% ↓ Self-‐Discipline ü 98.0% ü 94.7% -‐

↑ Job Performance Problems ü 98.1% -‐ -‐

↑ Substance Abuse Problems ü 99.4% -‐ -‐

↓ Self-‐Control ü 98.2% -‐ -‐

↓ Achievement Conformance ü 95.4% -‐ -‐





ü  If used properly, can it result in a reduced percentage of hired firefighters and EMS providers performing below standards?





q  Will these results remain significant over time? q  Can we improve our process to identify and remove an even higher

percentage of “Bad Hires”?


‘14 Data (Proba_on)

‘16 Data (2 Year FFs)

No Longer with AFD (6): 5.9% 5.9% -‐ 100% Separated during academy

-‐ 0% Separated during probaNon

-‐ 0% Separated a^er probaNon

Disciplined by PSO (1): 1.0% 1.0% -‐ Rated very highly by BC

Rated Below Average by BCs (9 & 15): 8.8% 14.7% -‐ Bad Driver Effect?

“Bad Hires” (16/102 & 22/102): 15.7% 21.6%


DQ’d in 2012 (17.8%)

2012 “Bad Hires” 2014 Data

Rank -‐ -‐ Psychologist RaNngs ü 99.9% Slower 1.5 Mile Run ü 99.4% ↑ PsyQ History of Work DifficulNes ü 99.9% -‐

↑ PsyQ History of Financial DifficulNes ü 99.8% -‐

↑ PsyQ History of Substance Use DifficulNes ü 99.9% -‐


DQ’d in 2012 (17.8%)



Rank -‐ -‐ -‐ Psychologist RaNngs ü 99.9% ü 98.9% Slower 1.5 Mile Run ü 99.4% ü 98.7% ↑ PsyQ History of Work DifficulNes ü 99.9% -‐ -‐

↑ PsyQ History of Financial DifficulNes ü 99.8% -‐ -‐

↑ PsyQ History of Substance Use DifficulNes ü 99.9% -‐ -‐


DQ’d in 2012


↑ Poorly Suited Candidate ü 98.2% -‐

↑ NarcissisNc Personality ü 98.2% -‐

↑ HosNlity ü 97.5% -‐

↓ Tolerance ü 99.9% -‐

↓ Amicability ü 99.4% -‐

↓ Integrity ü 99.9% -‐

↓ Achievement Independence ü 99.8% ü 98.4% ↓ Managerial PotenNal ü 99.5% -‐

↓ SocializaNon ü 98.4% -‐

↓ Intellectual Efficiency ü 96.1% ↓ Self-‐Discipline ü 98.0% -‐

↑ Job Performance Problems ü 98.1% -‐




Pre-Hire Personality Predictors (CPI) – 2016


DQ’d in 2012



↑ Poorly Suited Candidate ü 98.2% -‐ -‐

↑ NarcissisNc Personality ü 98.2% -‐ -‐

↑ HosNlity ü 97.5% -‐ -‐

↓ Tolerance ü 99.9% -‐ -‐

↓ Amicability ü 99.4% -‐ -‐

↓ Integrity ü 99.9% -‐ -‐

↓ Achievement Independence ü 99.8% ü 98.4% -‐

↓ Managerial PotenNal ü 99.5% -‐ -‐

↓ SocializaNon ü 98.4% -‐ -‐

↓ Intellectual Efficiency ü 96.1% ↓ Self-‐Discipline ü 98.0% -‐ -‐

↑ Job Performance Problems ü 98.1% -‐ -‐




Pre-Hire Personality Predictors (CPI) - 2016


DQ’d in 2012 (17.8%)

“Bad Hires” (‘02-‐’11) 2014 Data

Rank -‐ -‐ Psychologist RaNngs Slower 1.5 Mile Run ü 99.9% ↑ PsyQ History of Work DifficulNes ü 99.9% ü 99.9% ↑ PsyQ History of Financial DifficulNes ü 99.8% ü 99.7% ↑ PsyQ History of Substance Use DifficulNes ü 99.9% -‐

↑ PsyQ History of Psychological DifficulNes ü 93.8%


DQ’d in 2012 (17.8%)

“Bad Hires” (‘02-‐’11) 2014 Data

“Bad Hires” (‘02-‐’12) ‘14 or ‘16 Data

Rank -‐ -‐ -‐ Psychologist RaNngs Slower 1.5 Mile Run ü 99.9% ü 99.9% ↑ PsyQ History of Work DifficulNes ü 99.9% ü 99.9% ü 99.8% ↑ PsyQ History of Financial DifficulNes ü 99.8% ü 99.7% ü 99.6% ↑ PsyQ History of Substance Use DifficulNes ü 99.9% -‐ -‐

↑ PsyQ History of Psychological DifficulNes ü 93.8% ü 97.3%


DQ’d in 2012

“Bad Hires” (‘02-‐’11) 2014 Data







DQ’d in 2012

“Bad Hires” (‘02-‐’11) 2014 Data

“Bad Hires” (‘02-‐’12) ‘14 or ‘16 Data

↑ Poorly Suited Candidate ü 98.2% ü 96.3% ü 98.8% ↑ NarcissisNc Personality ü 98.2% ü 95.3% ü 92.0% ↑ HosNlity ü 97.5% ü 99.4% ü 98.8% ↓ Tolerance ü 99.9% ü 99.0% ü 99.6% ↓ Amicability ü 99.4% ü 98.9% ü 98.7% ↓ Integrity ü 99.9% ü 99.9% ü 99.7% ↓ Achievement Independence ü 99.8% ü 96.3% ü 98.6% ↓ Managerial PotenNal ü 99.5% ü 96.6% ü 97.2% ↓ SocializaNon ü 98.4% ü 98.9% ü 99.4% ↓ Intellectual Efficiency ü 96.1% ü 95.6% ü 95.5% ↓ Self-‐Discipline ü 98.0% ü 94.7% ü 97.1% ↑ Job Performance Problems ü 98.1% -‐ -‐









ü  Will these results remain significant over time? q  Can we improve our process to identify and remove an even higher

percentage of “Bad Hires”? q  YES!!! – Partnership with Background Investigators, Staff Physician and

Academy Leadership…

Suggestions for Further Discussion: Pre-Hire Psychological Testing

� Invest in Pre-Hire Psychological/Behavioral Health assessments. • Evidence that this can work (at a relatively low cost): ~$300/applicant vs. ~$120K to train a “Bad Hire” = 1/400!

Percent of Applicants Iden_fied as “Bad Hires”

2002-‐2011 (109-‐114)

2012 (115 & 116)

No Longer with AFD 11.8%* 5.9% Disciplined by PSO 4.9%* 1.0% Rated Below Average by BC’s 16.2%* 8.9% DQ’d by Psychologist 2.0% 17.8% TOTAL 32.2%* 33.6%

Suggestions for Further Discussion: Pre-Hire Psychological Testing

� Invest in Pre-Hire Psychological/Behavioral Health assessments.

� Use research to identify the factors that are the best predictors for your department and to evaluate the effectiveness of your pre-hire assessments.

� Carefully select, educate, and provide feedback to the psychologist(s) you hire to perform these assessments.

� Coordinate with the psychologist and use background evaluations to identify and disqualify clearly high behavioral risk applicants.

� Control the civil service appeals process.

Final Thought…

Whenever possible, collect, analyze, and use whatever data is available to improve your department’s ability to identify and reduce

“Bad Hires!”

Contact Information

For additional information, a copy of the slides, or to discuss opportunities for collaboration:

Marc Kruse, Ph.D. (512) 294-8092 (cell)

[email protected]

abetting the vetting: pre-hire intervention programs annual meeting handouts/atmd/na… · abetting...

Documents