validity in psychiatry · child and adolescent mental health haukeland university hospital, bergen,...
Post on 15-Aug-2020
2 Views
Preview:
TRANSCRIPT
Validity in Psychiatry
Maj-Britt Posserud, MD, PhD
Child and Adolescent Mental Health
Haukeland University Hospital, Bergen, Norway
Validity
” validity is the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world.”
Source: Wikipedia
Reliability and Validity
© Nevit Dilmen.
Various forms of validity
• Construct validity “the degree to which a test measures what it claims, or purports, to be measuring.”
– Cronbach & Meehl (1955) » Group differences » Correlation matrices and factor analysis » Predictive value
• Content validity ”the extent to which a measure represents all facets of a given social construct”
• Criterion validity ”is a measure of how well one variable or set of variables predicts an outcome”
Validation
1. Define «gold standard» – Reference for construct validity
2. Construct your measure – content validity, face validity
3. Apply your measure – Internal validity, reliability (cronbachs alpha)
– Factor analysis - construct/content validity
4. Compare to other measures/outcomes – Measurement invariance (CFA, SEM)
– Convergent validity w gold standard
– Discriminant validity (differentiating groups)
– Prediction (criterion validity)
The validity of categories «Diagnostic categories defined by their syndromes should be regarded as valid only if they have been shown to be discrete entities with natural boundaries that separate them from other disorders» «valid—meaning “well-founded…sound…against which no objection can fairly be brought”» «At present there is little evidence that most contemporary psychiatric diagnoses are valid…. This does not mean, though, that most psychiatric diagnoses are not useful concepts. In fact, many of them are invaluable. But, because utility often varies with the context, statements about utility must always be related to context, including who is using the diagnosis, in what circumstances, and for what purposes.» «If no detectable discontinuities in symptoms are found in large tracts of the territory of psychiatric disorder, it is likely that, sooner or later, our existing typology will be abandoned and replaced by a dimensional classification»
Am J Psych, 2003. Robert Kendell, M.D., Assen Jablensky, M.D.
ADHD symptoms
Parental report in the BCS
ADHD – DSM-IV
• ADHD symptoms:
«present to a degree that is maladaptive and
inconsistent with developmental level»
«often fails to», «often loses», «often
fidgets»
–«clear evidence of clinically significant
(distress or) impairment»
–«not occur exclusively during»
–«not better accounted for by»
Where is the deviance?
Impairment
• Intrinsic requirement for diagnosis
• However, the same degree of social ineptitude will lead to varying degrees of impairment depending on context
The diagnosis is not constant!
• Although vital for diagnosis, unspecific and not related to the underlying taxon or trait
Important concepts in screening/criterion validity:
• Sensitivity • Screen positive sick/Sick
• Specificity • Screen negative healthy/Healthy
• Positive predictive value • Screen positive sick/Screen positive
• Negative predictive value • Screen negative healthy/Screen negative
• Diagnostic Odds Ratio – DOR • Correctly classified/incorrectly classified
• Reciever Operating Curves
Factor analysis
• PCA – principal component analysis – Extracts all variance, including error variance
• EFA – exploratory factor analysis
• CFA – confirmatory factor analysis – A structure is decided beforehand – data is «fitted» to model
» Measure of model fit
• SEM – structural equation modelling – Latent variables may also be independent variables
Latent traits, SEM and measurement error
Measurement invariance & item response theory
• Construct validity • A measure may measure difference thing in different
places
• In psychiatry we generally measure latent traits
Factor analysis and structural equation modelling
Item response theory
• The probability that a person with a given trait will score on a particular item
SDQ - the world´s most available questionnaire!
• African languages: • Afrikaans
• Arabic (+ D)
• Amharic
• Chichewa
• Igbo
• Somali
• Swahili
• Xhosa
• Yoruba
• Zulu
• Asian languages: • Azeri (+ D) • Bengali (+D) • Chinese (+D) • Dari • Farsi • Georgian • Gujarati • Hindi • Hmong • Indonesian • Kannada • Malayalam • Pashto • Thai • ….
SDQ in 8 countries: UK, Norway, Bangladesh, India, Yemen, Brazil, Russia
”parent SDQ symptoms scores were so high in rural Yemen that the population mean of non-disordered children was comparable to that of children with a disorder in Britain. The converse was true of the final notably anomalous population…the 26 Indian children with a disorder.”
Conclusions:
”The result was that none of these SDQ caseness indicators could be used to make meaningful estimates of prevalence across the non-British samples.” ”population-specific norms may be needed when estimating prevalence… the SDQ shows large cross-cultural reporting effects and cannot be assumed a priori to be a valid method for comparing prevalences cross nationally without recourse to population-specific norms.” ”hihglights the importance of using local cultural and linguistic knowledge when reading the DAWBA transcripts and interpreting responses to structured questions.” ” We therefore recommend that questionnaires are only used in cross-cultural comparisons when their cross-cultural equivalence has been empirically demonstrated. Doing so may require detailed diagnostic measurements that employ local and contextual knowledge in order to provide population-specific reference points for judging the performance of brief questionnaire measures.”
SDQ Congo
Kashala, E., Elgen, I., Sommerfelt, K., & Tylleskar, T. (2005). Teacher ratings of mental health among school
children in Kinshasa, Democratic Republic of Congo. European child & adolescent psychiatry, 14(4), 208-215.
1 - 28% 2 – 9.6% 3 – 6.7% 4 – 5.2% 5 – 4.9%
3 .55
8 .69
13 .61
16 .69
24 .75
5 .58
7 .38
12 .64
18 .64
22 .59
2 .80
10 .80
15 .80
21 .48
25 .69
6 .74
11 .67
14 .52
19 .48
23 .70
1 .67
4 .75
9 .77
17 .68
20 .72 Kashala, E., Elgen, I., Sommerfelt, K., & Tylleskar, T. (2005). Teacher ratings of mental health among
school children in Kinshasa, Democratic Republic of Congo. European child & adolescent psychiatry,
14(4), 208-215.
Caci, H., Morin, A. J., & Tran, A. (2015). Investigation of a bifactor model of the Strengths and Difficulties Questionnaire.
European child & adolescent psychiatry, 1-11.
Receiver Operating Curves AUC – Area Under the Curve
ADHD bifactor analysis
One general factor
Hyperactivity ADHD Impulsivity and inattention also load on separate domains
ICD-10 organisation
top related