development and evaluation of the global appraisal of individual needs (gain) validity measures...
TRANSCRIPT
![Page 1: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/1.jpg)
Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures
Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut Health Systems, Bloomington, IL
and Richard Lennox, Psychometric Technologies, Hillsborough, NC
Workshop at the Joint Meeting on Adolescent Treatment Effectiveness (JMATE), Baltimore, MD, March 28, 2006.
![Page 2: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/2.jpg)
The content of this presentations are based on treatment & research funded by the Center for Substance Abuse Treatment
(CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA) under contract 270-2003-00006 using
data provided by the CYT grantees (Nos. TI11317, TI11320, TI11321, TI11323, and TI11324). The authors thank Sarah
Knecht, Michelle White and the GAIN QA team for helping to identify common inconsistencies, Sandra McGuinness for the code
for the inconsistencies and Barth Riley of UI-C for Rasch code. The opinions are those of the author and do not reflect official
positions of the consortium or government. Available on line at www.chestnut.org/LI/Posters or by contacting Joan Unsicker at 720 West Chestnut, Bloomington, IL 61701, phone: (309) 827-6026, fax: (309) 829-4661, e-Mail: [email protected]
Acknowledgement
![Page 3: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/3.jpg)
CYT Cannabis Youth Treatment Randomized Field Trial
Sponsored by: Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA), U.S. Department of Health and Human Services
Coordinating Center:Chestnut Health Systems, Bloomington, IL, and Chicago, ILUniversity of Miami, Miami, FLUniversity of Conn. Health Center, Farmington, CT
Sites:Univ. of Conn. Health Center, Farmington, CTOperation PAR, St. Petersburg, FLChestnut Health Systems, Madison County, ILChildren’s Hosp. of Philadelphia, Phil. ,PA
![Page 4: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/4.jpg)
Design • Target Population: Adolescents with marijuana disorders
who are appropriate for 1 to 3 months of outpatient treatment.
• Inclusion Criteria: 12 to 18 year olds with symptoms of cannabis abuse or dependence, past 90 day use, and meeting ASAM criteria for outpatient treatment
• Data Sources: self report on the GAIN (Dennis et al 2003), collateral reports, on-site and laboratory urine testing, therapist alliance and discharge reports, staff service logs, and cost analysis.
• Random Assignment: to one of three treatments within site in two research arms and quarterly follow-up interview for 12 months, and long term follow up 30 to 42 months later.
Source: Dennis et al 2002, 2004
![Page 5: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/5.jpg)
Adolescent Cannabis Users in CYT were as or More Severe Than Those in TEDS*
100%
15 intake
85%
46%
26%
78%
26%
47%
26%
71%
0%
20%
40%
60%
80%
First usedunder age
Dependence Weekly ormore use at
PriorTreatment
% o
f A
dmis
sion
s .
CYT Outpatient(n=600) TEDS Outpatient (n=16,480)* Adolescents with marijuana problems admitted to outpatient treatment
Source: Tims et al, 2002
![Page 6: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/6.jpg)
Demographic Characteristics
62%
15%
55%50%
30%
83%
17%
0%
20%
40%
60%
80%
100%
Female Male AfricanAmerican
Caucasian Under 15 15 to 16 Singleparentfamily
Source: Tims et al, 2002
![Page 7: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/7.jpg)
Institutional Involvement
25%
87%
47%
62%
0%
20%
40%
60%
80%
100%
In school Employed Current CJInvolvement
Coming fromControlled
Environment
Source: Tims et al, 2002
![Page 8: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/8.jpg)
Patterns of Substance Use
9%17%
71%73%
0%
20%
40%
60%
80%
100%
Weekly Tobacco Use
WeeklyCannabis Use
Weekly AlcoholUse
Significant Timein ControlledEnvironment
Source: Tims et al, 2002
![Page 9: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/9.jpg)
Multiple Problems are the NORM
86%
37%
12%
25%
61%
60%
66%
83%
83%
0% 20% 40% 60% 80% 100%
Any Marijuana Use Disorder
Any Alcohol Use Disorder
Other Substance Use Disorders
Any Internal Disorder
Any External Disorder
Lifetime History of Victimization
Acts of Physical Violence
Any (other) Illegal Activity
Three to Twelve Problems
Self-Reported in Past Year
Source: Dennis et al, 2004
![Page 10: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/10.jpg)
Co-occurring Problems are Higher for those Self-Reporting Past Year Dependence
71%
57%
25%
42%
30%37%
22%
5%
13%
22%
0%
20%
40%
60%
80%
100%
Health ProblemDistress*
Acute MentalDistress*
AcuteTraumaticDistress*
AttentionDeficit
HyperactivityDisorder*
ConductDisorder*
Past Year Dependence (n=278) Other (n=322)
Source: Tims et al., 2002 * p<.05
![Page 11: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/11.jpg)
Validity Measures• Number of Inconsistencies –Count of 49 paired items
consistently answered by over 90% of the clients, but that are inconsistent
• Denial/Misrepresentation – Sum of staff rating over 8 sections on a scale of 0-no problem, 1-estimating, 2-misunderstanding, 3-denial, 4-misrepresentation
• Context Effect – staff report of problems that might effect the interview (e.g.., someone present, interruptions, in juvenile justice setting)
• Proportion of Missing Data on 86 Items used in the GAIN’s core 10 Change measures: Substance Frequency Scale, Current Withdrawal Scale, Substance Problem Scale, Health Problem Scale, Emotional Problem Scale, Recovery Environment Risk Index, Social Risk Index, Illegal Activity Scale, Training Activity Scale and the Employment Activity Scale
![Page 12: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/12.jpg)
Validity Measures (Continued)This last two measures are based on theresidual (actual vs expected answers) of On the 123 items of the GAIN’s 4 mainpsychopathology and psychopathy scales. They are based the outfit and infit statistics under the Rasch (1960)measurement model and are reported inlogits• Atypicalness a measure of endorsing high
severity items without first endorsing the typical prior items (e.g.., suicide without depression)
• Randomness a measure of answers that are more random than expected on the GAIN’s 4 main psychopathology and psychopathy scales
N
yt
i ni2
Atypicalness
)1( nini
ninini
pp
pxy
Residual
N
inini
nini
N
i
pp
px
1
2
1
))1((
)(
Randomness
Where yni is the observed response of person n to item i and pni is the probability of a correct response for person n on item i.
![Page 13: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/13.jpg)
Correlation of Validity Measures
Bold indicates p < .05.
Denial/Misrepr. 0.07 -- -- -- --
Context Effect 0.10 0.31 -- -- --
%Missing Data -0.06 0.03 0.05 -- --
Atypicalness 0.03 0.20 0.05 -0.02 --
Randomness 0.00 0.24 0.14 -0.04 0.57
Inconsistencies -- -- -- ----
Den
ial/
Mis
repr
.
Con
text
Eff
ect
%M
issi
ng D
ata
Aty
pica
lnes
s
Ran
dom
ness
Inco
nsis
tenc
ies
----
--
--
--
--
While there is someoverlap, for the mostpart these measures
capture different aspects of validity
![Page 14: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/14.jpg)
Trichotomization of Validity Measures
• Like all GAIN scales we trichotomized the validity measures into low/mod/high range to help line staff interpret them.
• Because they were close to normally distributed, we divided Inconsistencies, Atypicalness and Randomness into: Low 0-50%, Mod 51-90% and High 91-100%
• Because they were sharply right skewed, Missing data, Denial/misrepresentation and Context effects were divided into Low 0%, Mod 1-90%, and High 91-100%
![Page 15: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/15.jpg)
Overview of Validation Test Results
Validity Measures
Denial/Misrep Rating
Fal
se N
egat
ives
Rel
ativ
e T
o U
rin
e at
Int
ake
X
Missing Data
Context
Inconsistencies
Randomness (aka infit)
Bia
s 3
Mon
th o
utco
mes
X
X
X
Atypicalness (aka outfit)C
onst
ruct
Val
idit
y at
Int
ake
X
X
X
X
X
X
Tes
t-R
etes
t at
Int
ake
X
X
X
X
X
X
Inte
rnal
Con
sist
ency
at
Int
ake
X
X
X
X
X
X: Continuous or trichotomous version of validity measure is a statistically significant (p<.05) predictor of worse values on the criterion in this column
![Page 16: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/16.jpg)
Internal Consistency Results
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Ave
rage
Cro
nbac
h’s
Alp
ha*
Low 0.87 0.87 0.87 0.87 0.87 0.87 0.87
Mod 0.88 0.88 0.86 0.88 0.86 0.87 0.87
High 0.89 0.84 0.80 0.83 0.82 0.84
Denial/Misrep Rating
Missing Data Context Inconsistecies Randomness Atypicalness Average
*average alpha across Substance Problem Scale, Internal Mental Distress Scale, Behavior Complexity Scale & Crime/Violence Scale
Inconsistencies are the best predictor of low alpha
![Page 17: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/17.jpg)
Test-Retest Results
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Ave
rage
Tes
t-R
etes
t Rho
*
Low 0.68 0.66 0.67 0.66 0.68 0.70 0.68
Mod 0.64 0.64 0.65 0.69 0.64 0.62 0.65
High 0.61 0.59 0.65 0.56 0.59 0.60
Denial/Misrep Rating
Missing Data Context Inconsistecies Randomness Atypicalness Average
*average test-retest rho over 2-14 days for days of alcohol use, marijuana use, any substance use, and number of cannabis disorder symptoms
Randomness is the best predictor of low test-retest Rho
![Page 18: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/18.jpg)
Correlations with Intake Variables
Bold indicates p < .05.
Spearman Rho
Day
s of
Any
AO
D u
se
Day
s of
Alc
ohol
Use
Day
s of
M
arj.U
se
Day
s of
AO
D P
robl
ems
Pas
t M
onth
Pro
blem
s
Day
s of
Arg
uing
/Vio
lenc
e
Day
s of
Ille
gal A
ctiv
ity
Day
s of
Ill
ega
l Act
ivity
for
$
Denial/Misrep -0.06 -0.07 -0.04 -0.10 -0.13 -0.02 -0.04 0.00
Missing Data 0.00 0.02 -0.01 -0.02 0.03 -0.03 0.05 0.07
Context 0.04 -0.01 0.06 -0.09 -0.02 -0.04 0.00 0.00
Inconsistencies -0.01 -0.01 -0.04 0.02 -0.02 -0.05 -0.14 -0.09
Randomness -0.17 -0.12 -0.14 -0.25 -0.23 -0.05 0.01 0.09
Atypicalness 0.02 0.02 0.03 -0.03 0.02 0.11 0.17 0.20
Randomness on symptoms generally predicts lower than expected values
Atypicalness predicts higher than
expected values
![Page 19: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/19.jpg)
Staff Ratings of Denial/Misrepresentation Predict False Negatives Relative to Urine Screens
6%
15%
5%
0%
2%
4%
6%
8%
10%
12%
14%
16%
Denial/Misrepresentation
LowModHigh
Only significant predictor of False Negatives, OR=4.0, 95% CI (2.03, 7.96)
But not all Denial/
Misrepresentation is
about drug use
![Page 20: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/20.jpg)
Predicting 3-Month Outcomes
• Predicted 3-month variables with intake only and in second step, added the validity measures
• Dependent Variables: Substance Frequency, Substance Problems, Emotional Problems, Recovery Environment, Social Risk and Illegal Activity
• Context and Inconsistencies had a small significant positive relationship with Recovery Environment Risk at 3 months
• Randomness had a small significant positive relationship with Substance Frequency and Illegal Activity
![Page 21: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/21.jpg)
Limitations
• We had very little missing data, denial/ misrepresentation, and false negatives in the data from this multi-site clinical trial
• This data set was for outpatient and limited in severity and diversity
• We plan to replicate the analyses with several larger data sets that are more diverse in terms of clinical severity, geography, demographics, level of care and type of service providers
![Page 22: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/22.jpg)
Conclusions
• The 6 GAIN validity measures are good markers for predicting problems with internal consistency, reliability, and validity
• Even where there were problems, self report was still generally reliable and valid
• The small correlations between measures and differences in what they predicted demonstrate that they are measuring different facets of the problem
• Having developed metrics for identifying problem cases, the next step is to develop interventions to reduce the likilihood of these problems.
![Page 23: Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut](https://reader035.vdocuments.mx/reader035/viewer/2022062714/56649d215503460f949f74bb/html5/thumbnails/23.jpg)
References
• Dennis, M. L., Godley, S. H., Diamond, G., Tims, F. M., Babor, T., Donaldson, J., Liddle, H., Titus, J. C., Kaminer, Y., Webb, C., Hamilton, N., & Funk, R. (2004). The Cannabis Youth Treatment (CYT) Study: Main Findings from Two Randomized Trials. Journal of Substance Abuse Treatment, in press
• Dennis, M. L., Titus, J. C., Diamond, G., Donaldson, J., Godley, S. H., Tims, F., Webb, C., Kaminer, Y., Babor, T., Roebeck, M. C., Godley, M. D., Hamilton, N., Liddle, H., Scott, C., & CYT Steering Committee. (2002). The Cannabis Youth Treatment (CYT) experiment Rationale, study design, and analysis plans. Addiction, 97, 16-34.
• Dennis, M. L., Titus, J. C., White, M. K., Unsicker, J., & Hodgkins, D. (2003). Global Appraisal of Individual Needs: Administration Guide for the GAIN and Related Measures. Bloomington, IL: Chestnut Health Systems. Retrieved from http://www.chestnut.org/li/gain .
• Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedogogiske Institut. (Republished Chicago: The University of Chicago Press: 1980).
• Tims, F. M., Dennis, M. L., Hamilton, N., Buchan, B. J., Diamond, G. S., Funk, R., & Brantley, L. B. (2002). Characteristics and problems of 600 adolescent cannabis abusers in outpatient treatment . Addiction, 97, 46-57.