an experimental study of child welfare worker turnover
DESCRIPTION
An Experimental Study of Child Welfare Worker Turnover. Nancy S. Dickinson, University of Maryland [email protected] John S. Painter [email protected]. Child Welfare Staff Recruitment and Retention: An Evidence Based Training Model. Study Objectives - PowerPoint PPT PresentationTRANSCRIPT
An Experimental Study of Child Welfare Worker Turnover
Nancy S. Dickinson, University of Maryland
[email protected] John S. Painter
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Child Welfare Staff Recruitment and Retention: An Evidence Based Training ModelStudy Objectives• Determine the feasibility of using an
experimental design to study training outcomes
• Understand the impact of worker perceptions on their intent to leave child welfare employment
• Study the effectiveness of the intervention on worker retention
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Retention
External Environment• Agency’s public
image• Awareness of jobs
Agency Climate• Shared mission• Affirmation &
recognition• Shared authority• Growth &
advancement• Org commitment
Worker Characteristics• Desire to help• Self-efficacy• Depersonalization• Education
Supervision• Practice support• Emotional support• Team support
The Work• Role clarity• Role expectations• Workload
Influences on Recruitment, Selection and Retention
Recruitment
Selection
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Intervention ComponentsRecruitment Selection Retention
•Poster •Realistic Job Preview DVD An Invitation to Choose
•A Supervisor’s Guide to Retention
•Flyers •Competency Based Selection Process
•A Director’s Guide to Retention
•Custom Brochures •Selection Training •Retention Training•2 30-second PSA’s •Retention Toolkit•Slide Presentation •Technical Assistance•Recruitment Training
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Research Questions• Do workers in the intervention
counties show statistically significant differences from those in the control counties on relevant survey scales?
• Does child welfare worker retention improve in the intervention counties compared with the control counties?
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Procedures• Random assignment of county child
welfare agencies to 17 intervention and 17 control groups
• 33 project counties participated in data collection activities (1 agency withdrew after a year)
General DesignIntervention: R O1 X
O2
Comparison: R O1O2
R = random assignmentO = data collection (or observation)X = intervention or treatment National Human Services Training Evaluation Symposium
Cornell University, June 15, 2011
Instruments• Online worker survey administered 5
times to all project child welfare workers between 6/1/05 and 6/1/08
• Human Resources Database gathered employment information on all project workers between 12/1/04 and 9/1/08
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Worker Survey• 17 scales validated using reliability
analysis and confirmatory factor analysis
• Average response rate of 47% (45% - 48%) across 5 waves of delivery to an average sample of 831 workers (731-944)
• Waves 1 & 2 were pre-intervention; waves 4 & 5 were post-intervention.
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Respondent DemographicsWave 1 Wave 2 Wave 3 Wave 4 Wave
5Total 356 354 386 422 446 Gender
Female 87.1% 86.7% 88.3% 89.2% 87.5% Male 12.9% 13.3% 11.7% 10.8% 12.5%Race
African-American
23.9% 17.8% 24.4% 26.1% 24.4%
European-American
68.3% 70.8% 63.5% 63.7% 66.1%
Other 7.9% 11.3% 12.2% 10.2% 9.5%Age
Average 38 39 38 39 39St. Dev 10.5 10 10.5 10.5 11
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Demographics, ContinuedWave 1 Wave 2 Wave 3 Wave 4 Wave
5Total 356 354 386 422 446Degree Type
Bachelor 47.2% 45.1% 47.4% 47.5% 46.8%Master 7.6% 7.4% 6.9% 7.6% 6.4%BSW 33.4% 33.7% 31.7% 29.4% 30.7%MSW 11.7% 13.8% 14.0% 15.5% 16.1%Missing (4.2%) (7.6%) (9.3%) (6.6%) (6.4%)Caseload
Average Number of Families per Month
14 13 13 13 12
St. Dev 7.6 7.5 7.8 7.6 7.5
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Data Analyses• Multi-level regression analysis
–Scales are compared pre-post intervention
• Survival analysis –Days employed & status at end of
study (exit vs. no exit)
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Statistical Comparisons for Survey Scales• Four primary comparisons were
made:– Intervention vs. control post-training
• Individual level• County level
– Pre vs. post training intervention group only• Individual level• County level
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Overview of Survey ResultsScale Indiv.
Interv. Vs. ControlCnty.
Interv. Vs. Control
Indiv. Pre. Vs. Post
Cnty. Pre. Vs. Post
S1 Depersonalization * *** **S2 Desire to help *** **S3 Self Efficacy ** *** ***S4 Workload ***S5 Role Clarity * *** *S6 Role Expectations S7 Supervisor: Practice Support ** *S8 Supervisor: Team Support *S9 Supervisor Emotional Support S10 Organizational Commitment **S11 Agency’s Negative Image S12 Agency Affirmation * **S14 Shared Mission *** ***S15 Shared Authority S16 Growth & Advancement Opportunities *S17 Intent To Leave (lower scores indicate lower intent to leave) * ** * *
* P < .05, ** p < .01; *** p < .001
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Impact of Intervention on Turnover: Data and Sample• HR Database used by all project
counties– Internet accessible– Interactive database application
• In 9/08, analysis file of 877 workers hired after January 1, 2004– 485 workers from control counties– 392 from intervention counties
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Worker Demographics for Original and Propensity Matched SamplesOriginal Matched
Intervention Control Intervention Control
Total 392 485 318 318Degree Type N % N % N % N %
No SW Degree 236 60.2% 195 40.2% 165 51.9% 156 49.1%BSW or MSW 156 39.8% 290 59.8% 153 48.1% 162 50.9%
Previous ExperienceNone 73 18.6% 72 14.8% 65 20.4% 56 17.6%Indirect 66 16.8% 67 13.8% 35 11.0% 58 18.2%Direct 253 64.5% 346 71.3% 218 68.6% 204 64.2%
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Analysis and Results• Cox regression survival analysis
assessed the impact of the intervention on undesirable exits
• Effect of the intervention is statistically significant (p<.05)– 27% of control group sample
experienced an undesired exit– 17% exit in the intervention group
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
6606306005705405104804504203903603303002702402101801501209060300-30
Days On Job
1.0
0.9
0.8
0.7
0.6
0.5
0.4
Cum
Sur
viva
lSurvival Function
InterventionComparison
Time varying intervention variable
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Type of Exit Post InterventionGroup Type of Exit Frequency PercentControl No Exit 183 67.8
Undesired Exit 74 27.4Promoted 2 0.7Transferred 10 3.7Other 1 0.4Total 270 100
Intervention No Exit 158 70.9Undesired Exit 39 17.5Promoted 6 2.7Transferred 17 7.6Other 3 1.3Total 223 100
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Summary• A rigorous research methodology can
be used to test the effectiveness of a training intervention.
• Undesired exits by child welfare workers can be slowed significantly because of increased skills and behaviors of supervisors and managers.
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Limitations• Absence of statewide employee
database limits quality of data.– Some concern that project database was
used inconsistently– Cannot track workers across counties to
determine if worker left the profession or the agency
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
What Worked Well• Recruiting counties thru site visits • Random assignment• Providing counties with data on
turnover • Longitudinal design• Control group• Lots of personal contact with
counties• HR dbase data proved key• Web surveys were very efficientNational Human Services Training Evaluation Symposium
Cornell University, June 15, 2011
Think Twice…• Number of counties in study• Number of times surveyed• Web reports
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Unexpected Challenges• Data management! A beast…• Some counties were inconsistent in
use of HR dbase• Inconsistent response to surveys left
gaps in data• Collecting baseline data before
intervention was finalized
National Human Services Training Evaluation SymposiumCornell University, June 15, 2011
Acknowledgement• This study was supported by the U.S.
Children’s Bureau (Grant No. 90CT0114) as part of the project Child Welfare Staff Recruitment and Retention: An Evidence-Based Training Model.
THANKSNational Human Services Training Evaluation Symposium
Cornell University, June 15, 2011