predicting success in job corps - alexander eschbach

Predicting Success in Job Corps Students

What Variables Create a Completer?Chicago Job Corps Longitudinal Study

Alexander Adam Eschbach, Ph.D.Center Mental Health Consultant

Paul Simon Chicago Job Corps CenterOperated by Management and Training Corporation

FuturePrese

ntPast

Predicting SuccessPresentation Structure Inspecting the data first…

Sample size Gender, race Entry drug testing Educational characteristics Approach to psychological symptom

checklist

Past

Predicting SuccessPresentation Structure How we are using the data…PastPresent Future

Past

Present

Future

Predicting SuccessPresentation StructureHow we are using the data… Past

Examining trends in student characteristics

Identifying factors that appear to predict success

Past

Predicting SuccessPresentation StructureHow we are using the data… Present

Using data collected today to help students today

Interventionally relevant data (e.g., suicide risk assessment for students endorsing suicidal thoughts)

Present

Predicting SuccessPresentation StructureHow we are using the data… Future

Regularly using mathematical formula to identify “at-risk” students

Developing “tailor-made” interventions to increase student retention

Employing “evidence-based” methods to increase success

Future

Predicting SuccessPresentation StructureWarnings/caveats about using/interpreting the

data… Reflects data from Chicago Job Corps only Every center has its own unique geographical

location (urban vs. rural) Unique racial, ethnic background Differences in academic and vocational

programs Cannot assume data presented here will apply

to another Job Corps center Necessary to test this prediction model at

other centers

Past

Predicting SuccessAcknowledgements

Many thanks to… MTC Research Institute

Carl Nink, Director of the MTC Research Institute Rob Olding, Ph.D., Statistician and Dean of University of

Phoenix Management Training Corporation (MTC)

Christina Hunter , MTC Director of Operations Anita Sharp, MTC Vice President, Central Region

Chicago Job Corps Administration and Colleagues Bryan Mason, Center Director, Paul Simon Chicago Job Corps Gemma Ross, Health and Wellness Manager, Chicago Job Corps

National and Regional Mental Health Consultants Valerie Cherry, Ph.D., Lead National Mental Health Consultant Helena Mackenzie, Ph.D., Region V Mental Health Consultant

Longitudinal Study

Data from nearly 3,000 CJC students

Separated from March 2004 – December 2009

Only students who completed SCL-90-R

Only students with complete data

Past

Longitudinal StudySCL-90-R Information 90-item checklist for physical, behavioral,

and psychological complaints Every new student completes SCL-90-R on

second day People with 6th grade reading level can

understand questions Literature that shows this checklist has been

used with various ethnic and racial groups “Face-valid,” “What you see is what you

get”

Past

Longitudinal StudyWhy look at this data? Problem with “revolving door” noticed

in the early 2000s 350 – 375 positions, requiring 1 – 2

years to “complete” Averaging 600 new students a year Huge attrition (loss of students) due to

Drug use Violence Resignation AWOL Medical problems

Past

Longitudinal StudyWhy look at this data? Huge cost to Department of Labor

secondary to “revolving door” Considerable stress on CJC staff

Much effort to orient new students Much disappointment when student

does not finish program Minimizing negative experiences for

students due to failure Ultimately identifying “at-risk”

students early to improve retention and chance for success

Past

Longitudinal StudyBeginning Assumptions(A Priori Hypotheses) Probably a combination of factors

would predict success Math, reading ability Prior drug use (“positive on entry”) Psychological issues (depression, etc.) Behavioral issues (self-discipline,

persistence) Age Gender

Past

2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

14

Longitudinal StudySample Information

1388; 48%1499; 52%

Gender

MaleFemale

Past


15


101; 4%359; 14%

522; 21%

443; 18%

376; 15%

267; 11%

169; 7%

115; 5%

102; 4% 33; 1%3; 0%1; 0%

Age at Separation

16 1718 1920 2122 2324 2526 27

Past


16


1845; 64%

713; 25%

288; 10% 24; 1%

Racial African-Amer-icanLatinoCaucasianOther

Past


17


602; 24%124; 5%

544; 22%1043; 42%

35; 1%93; 4% 27; 1% 16; 1%1; 0%

Separation Type

Disciplinary Medical AWOLCompleter Parental Withdrawal ResignationRegular Transfer AT Transfer Adminis. Separation

Past


18


599; 21%

2287; 79%

Drugs on Entry to Program

PositiveNegative

Past


19


351; 59%

248; 41%

Male vs. Female Positive on Entry

MaleFemale

Z = 5.78, p < 0.01 (tw0-tailed t-test). Females significantly lesslikely to enter the center with a “positive” on drug testing.

Past

Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”)

2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul

Simon MTC Chicago Job Corps 20

01020304050607080

Disciplinary, 50.8

Disc Viol Mand, 34.7

Disc Viol, 23.1Reg Trans,

22.2Disc Viol

Nonmand, 18.7AWOL, 15.6

Resignation, 15.1

Ordinary Separation,

12.4Med Sep w/ Reinstate, 9.4

Parent Withdraw,

8.6MSWR Final Close, 5.6

Fraudulent, 0

Positive on Entry % by Separation Type

Past



Simon MTC Chicago Job Corps 21

Reading Math480490500510520530540550560

Ordinary SeparationDisciplinaryParent WithdrawMed Sep ReinMSWR FinalResignationAWOLDisc NonmandDisc MandatoryDisc Drugs

Past

TABE Scores by Separation Type


Weird, consistent trends from SCL-90-R data Reflected Chicago Job Corps students’

unique response pattern A pattern noticed before during dissertation

research A subset of male --- but sometimes female ---

students who selected zero or just a few SCL-90-R items were often found to be “positive on entry” for drug use. These were “Deniers” (students who selected 0 – 3 SCL-90 items) “There’s nothin’ wrong with me, but, oh, I

use weed a lot.” These students were often separated for

drug use or violence.

Past


Generally speaking, it appeared that Ordinary separation students (“Completers”) endorsed a moderate number of items as a group. They were called “Reporters” (Students who selected 4 - 37 SCL-90-R items)

Past


Students who endorsed many SCL-90-R items were often separated for medical or psychiatric reasons. These were called “Exaggerators” (Students who selected 38 – 90 SCL-90-R items.)

Past

Longitudinal StudyUnivariate Look at the Data(“One Thing at a Time”)

Using the SCL-90-R to help students in the “Here and Now.”

SCL-90-R “Top 10” Items Endorsed by Chicago Job Corps students…

Past


10. Trouble concentrating. 9. Awakening in the early

morning. 8. Pains in lower back. 7. Trouble remembering

things. 6. Having to check and

double-check what you do.

Past


5. Trouble falling asleep. 4. Feeling easily annoyed or

irritated. 3. Worrying too much about

things. 2. Headaches.

Past


1. Feeling that most people cannot be trusted.

Past

29

Item Content En-dorsed

%

Not bother

-ed

A little bit

Moder-ately

Quite a bit

Extreme-ly

Sample N

En-dorsed

1. Trust 64% 1033 751 367 345 384 2880 1847

2. Headache 61% 1123 995 387 291 84 2880 1757

3. Worry 54% 1331 780 341 226 205 2883 1552

4. Annoyed 52% 1377 804 321 231 142 2875 1498

5. Memory 48% 1479 905 237 198 60 2879 1400

6. Double-check 47% 1521 810 287 158 105 2881 1360

7. Sleep trouble 41% 1705 561 224 207 186 2883 1178

8. Trouble concentrating 39% 1764 675 222 125 98 2884 1120

9. Back pain 38% 1786 522 246 204 121 2879 1093

10. Early awakening 38% 1778 575 246 151 122 2872 1094

Top 10, Most Frequently Endorsed SCL-90-R Items

Chicago Job Corps Students 3/2004 – 12/2009

Past


Simon MTC Chicago Job Corps


30


Curious about the least selected items?

SCL-90-R “Least Popular” Items Endorsed by Chicago Job Corps students…

Past


31


81. Shouting or throwing things.

82. Trembling. 83. Heavy feelings in your

arms or legs. 84. Having thoughts that are

not your own. 85. Feeling afraid to go out of

your house alone.

Past


32


86. Thoughts of ending your life.

87. The idea that someone else can control your thoughts.

88. Feeling afraid you will faint in public.

89. A lump in your throat.

Past


33


90. Hearing voices that others do not hear.Only 128/2883 (4%) of

students selected this item.

Past

34

Bottom 10, Least Frequently Endorsed SCL-90-R ItemsChicago Job Corps Students 3/2004 – 12/2009

Item Content En-dorsed %

Not bother

-ed

A little bit

Moder-ately

Quite a bit

Extreme-ly

Sample N

En-dorse

d

1. Shouting, throwing

10% 2592 202 42 27 19 2882 227

2. Trembling 10% 2591 197 43 28 16 2875 284

3. Heavy extremities

10% 2602 186 46 30 14 2878 276

4. Thoughts not your own 9% 2625 168 47 24 15 2879 254

5. Afraid to leave house 8% 2654 142 41 29 15 2881 227

6. Thoughts of ending life 7% 2670 140 41 17 18 2886 216

7. Thoughts controlled 7% 2680 118 33 26 18 2875 195

8. Afraid of fainting 7% 2680 122 39 21 12 2874 194

9. Lump in throat 6% 2714 106 31 16 16 2883 169

10. Hearing voices 4% 2755 81 26 9 12 2883 128



Past


35

Longitudinal Study

Suicide Info (Center for Disease Control, Atlanta) In 2009, 13.8% of U.S. high school

students reported that they had seriously considered attempting suicide during the 12 months preceding the survey; 6.3% of students reported that they had actually attempted suicide one or more times during the same period.

Present


36


SCL-90-R Data in the “Here and Now” Students endorsing Item #15 on the SCL-90-R

(Thoughts of ending your life) are seen within 24 hours of arriving on center

Suicide Risk Assessment conducted by CMHC and/or Psychology Extern (Practicum Student)

Base rate of suicidal ideation reported on SCL-90-R lower at CJC than reported base rate by CDC for same age group (13.8% U.S. high school population; 216/2887 [7.0 %] CJC students endorsed Item #15)

Present


37


SCL-90-R Data in the “Here and Now” In our sample, Item #15 (Thoughts of

ending your life) was ranked 86th out of 90 items.

One of the least frequently endorsed items.

When it was endorsed, more than 2/3 of the students reported they misread the item, made a mistake, or they were reporting a former problem and not current suicidal thinking.

Present


Summary of the Univariate Findings “Positive on Entry” is important TABE reading and math scores are

Important Comparing “Deniers” to “Exaggerators”

to “Reporters” is important Is gender important? Other variables to predict success?

Past

Future

Preliminary Conclusions

Mathematical model confirms our original scientific guess (hypothesis) that there are identifiable factors predict success

Past

Future

Next StepsWhere did we go from here? Further data analyses Use logistic regression to introduce different

variables that could predict “group membership” “Plug in” TABE scores, Positive on Entry, SCL-90-R

data, and other possible variables into a formula to calculate a “number”

The greater that “number” is more than 0 (zero), the greater the odds that the student will be in the Successful Completer group

If the calculated “number” is less than 0 (zero), the lower the number, the more likely that the student will be in the Unsuccessful/Noncompleter group

Future

Past

Next StepsWhere did we go from here? Various separation types divided into

Unsuccessful (Noncompleters), Successful (Completers), and Neutral outcomes. Completion, Ordinary Transfers = Successful Resignation, Disciplinary, AWOL = Unsuccessful Medical separation = Neutral

Use univariate observations to “plug into” multivariate (“Logistic Regression”) analysis

Concept is that outcomes are explained by the complex interaction of several variables

Future

Next StepsWhere did we go from here?Variable Coefficien

tApprox.

SEWald Chi-

Sq.P - value

Intercept -7.212000 0.67 116.3 < .001Pos. on Entry -1.183000 0.13 80.4 < .001Highest Grade .2477700 0.03 52.5 < .001TABE Math/100 .4427600 0.13 12.5 < .001SCL-90-R D, R, E

.2914200 0.09 10.0 0.002TABE Reading/100

.3003800 0.13 5.3 0.022Gender .2164100 0.11 4.2 0.04

Building the Logistic Regression (Mathematical) Model

Future

Next StepsWhere did we go from here?Building the Logistic Regression (Mathematical) Model Positive on entry Highest grade achieved Initial TABE math score Initial TABE reading score SCL-90-R category

Denier (0 – 3 items) Reporter (4 – 37 items) Exaggerator (38 – 90 items)

Gender

Future

44

Next StepsWhere did we go from here?Building the Logistic Regression

(Mathematical) Model Predicted Outcome

Observed Outcome

Noncompleters

Completers

Noncompleters

666 276 942

Completers 322 481 803

988 757 1745Correctly predicted percent = 65.73% ≈ 66%. False positives = 36.46%.False negatives = 32.59%. Area under the ROC curve, c = 0.7158. Sensitivity = 59.9%. Specificity = 70.7%. (Random samples of larger data yielded same results.)

Future



Next StepsWhere did we go from here?Building the Logistic Regression (Mathematical) Model The Logistic Regression Model (the equation

[formula] produces a number. If we want to predict better than “chance,” we started by using 0.5 as the “cut-off” score. The greater than 0.5 that a student scores, the greater the odds that he or she will be a Successful Completer. If the student scores much less than 0.5, the greater the odds that he or she will be an Unsuccessful Noncompleter.

Future

The logistic function is useful because it can take as an input any value from negative infinity to positive infinity, whereas the output is confined to values between 0 and 1. The variable z represents the exposure to some set of independent (predictor) variables, while ƒ(z) represents the probability of a particular outcome, given that set of explanatory variables. The variable z is a measure of the total contribution of all the independent variables used in the model and is known as the logit.

The variable z is usually defined as:

http://en.wikipedia.org/wiki/Logit

Z

Probability for Success Plotting Using Logistic Regression

48

So a student who was positive on entry (= 1), was male (= 1), had reached the 9th grade, received an Initial TABE Reading score of 490, an Initial TABE Math score of 507, and was in SCL-90-R Category 3 would have the following regression prediction equation beginning with the Intercept (mathematical constant):-7.212 + 1(-1.1830) + 1(.21641) + 9(.24777) + (490/100[.30038]) + (507/100[(.44276]) + 3(.29142) = -1.3577448. If -1.3577448 is plotted on the graph, it becomes apparent that the student would definitely have less than a 50-50 chance of completing the Job Corps program if we do not intervene and help him.

Future



Z

Plotting Chance for SuccessMale Student Example ( -1.3577448

)Less Than 50-50 Chance for

Success

Z = -1.3577448

50-50 Chance

50

In contrast, a student who was negative on entry (= 0), was female (= 0), had reached the 12th grade, received an Initial TABE Reading score of 650, an Initial TABE Math score of 700, and was in SCL-90-R Category 2 would have the following regression prediction equation beginning with the Intercept (mathematical constant):

-7.212 + 0(-1.1830) + 0(.21641) + 12(.24777) + (650/100[.30038]) + (700/100[(.44276]) + 2(.29142) = +1.39587. If +1.39587 is plotted on the graph, it becomes immediately apparent that the student has considerably more than a 50-50 chance of being a successful completer from Job Corps.

Future



Z

Plotting Chance for SuccessFemale Student Example

( +1.39587 )Greater Than 50-50 Chance for

Success

Z = + 1.39587

50-50 Chance

52

An extreme case of success would be a student who was negative on entry (= 0), was female (= 0), had reached the 14th grade, received an Initial TABE Reading score of 700, an Initial TABE Math score of 750, and was in SCL-90-R Category 2 would have the following regression prediction equation beginning with the Intercept (mathematical constant):-7.212 + 0(-1.1830) + 0(.21641) + 14(.24777) + (700/100[.30038]) + (750/100[(.44276]) + 2(.29142) = +2.26298. See the plot on the next page to see the marked increase in probability of success.

Future



Z

Plotting Chance for SuccessExtreme Female Student Success Example ( + 2.26298 )

Over 90% Chance for Success

Z = + 2.26298

50-50 Chance

2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon Chicago Job Corps

54

An extreme case of non-success would be a student who was positive on entry (= 1), was male (= 1), had reached the only the 8th grade, received an Initial TABE Reading score of 400, an Initial TABE Math score of 380, and was in SCL-90-R Category 1 would have the following regression prediction equation beginning with the Intercept (mathematical constant):-7.212 + 1(-1.1830) + 1(.21641) + 8(.24777) + (400/100[.30038]) + (380/100[(.44276]) + 1(.29142) = - 3.021. See the plot on the next page to see the marked decrease in probability of success.

Future

Z

Plotting Chance for SuccessExtreme Male Non-success Example ( - 3.021002 )

Less Than 5% Chance for Success

Z = -3.021002

50-50 Chance

Next StepsHow will the data help our students? What does the data mean?

Is ability to be “Negative on Entry” a sign of self-discipline?

Is “Highest Grade Achieved” a sign of persistence?

Is “Initial TABE Math score” a sign of problem-solving skills?

These factors appear most predictive of success.

FutureFuture

Next StepsHow will the data help our students? Early prediction of group

membership helps staff intervene In consultation with the CMHC and

Wellness personnel, academic, vocational, and residential staff can work together to develop a plan to support/intervene with a student who presents the risk factors of students prone to leave before completion of the program.

FutureFuture

PREDICTING SUCCESS IN JOB CORPS STUDENTS

What Variables Create A Completer?

Chicago Job Corps Longitudinal Study

Questions?

Comments?

Future

predicting success in job corps - alexander eschbach

Documents