Slide 1

A New Look at Evidence Based Approaches in ADHD Assessment EVIDENCE Thomas K. Pedigo Ed.D., NCSP Vann B. Scott, Jr., Ph.D. Ron P. Dumont Ed.D., NCSP Slide 2 DISCLOSURES THOMAS K. PEDIGO, ED.D., NCSP DIRECTOR SAVANNAH CHILD STUDY CENTER, SAVANNAH GEORGIA. PLEASE NOTE THAT DR. PEDIGO IS THE CO-AUTHOR OF THE PEDIATRIC ATTENTION DISORDERS DIAGNOSTIC SCREENING SYSTEM AND CO-OWNER OF TARGETED TESTING INC. WHICH ARE REFERENCED IN THIS PRESENTATION. VANN B. SCOTT JR., PH.D. ASSOCIATE PROFESSOR OF PSYCHOLOGY ARMSTRONG ATLANTIC STATE UNIVERSITY, SAVANNAH GEORGIA. PLEASE NOTE THAT DR. SCOTT HAS SERVED AS A RESEARCH CONSULTANT TO TARGETED TESTING INC. DURING THE DEVELOPMENT OF THE PEDIATRIC ATTENTION DISORDERS DIAGNOSTIC SCREENING SYSTEM. RON P. DUMONT, ED.D., NCSP ASSOCIATE PROFESSOR OF PSYCHOLOGY & DIRECTOR OF THE DOCTORAL TRAINING PROGRAM IN SCHOOL PSYCHOLOGY AT FAIRLEIGH DICKENSON UNIVERSITY, TEANECK NEW JERSEY. PLEASE NOTE THAT DR. DUMONT DOES NOT HAVE PERSONAL OR FINANCIAL INTEREST IN THE PEDIATRIC ATTENTION DISORDERS DIAGNOSTIC SYSTEM OR WITH TARGETED TESTING INC. WHICH ARE REFERENCED IN THIS PRESENTATION. Slide 3 Dr. Ron P. Dumont Slide 4 Slide 5 Suppose a test was used as a screening measure on a population of 1000 children in which 4% (40) of the children have ADHD, and that test gives an abnormal score for 90% of the children with ADHD (i.e., sensitivity) and gives a normal score for 90% of children without ADHD (i.e., specificity), Slide 6 with disorderSensitivity: proportion of children with disorder who received abnormal test scores Slide 7 without disorderSpecificity: proportion of children without disorder and who received normal test scores Slide 8 Positive Predictive Power (PPP): probability that one who receives an abnormal test score is correctly classified Slide 9 Negative Predictive Power (NPP): probability that one who does not have an abnormal test score is not classified Slide 10 Slide 11 CONSIDERATION OF BASE RATES A GIVEN TESTS PREDICTIVE POWER MUST BE CONSIDERED IN LIGHT OF THE GIVEN BASE RATE IN THE POPULATION FROM WHICH IT IS USED. A TEST WITH.90 SENSITIVITY & SPECIFICITY 100%.90 50%.40 BASERATE MAXIMUM POWER BASERATE MAXIMUM POWER BASERATE 04% MAXIMUM POWER.20 Slide 12 POSITIVE & NEGATIVE PREDICTIVE POWER COMBINED ADHD AND NON-CLINICALS N=200 SINCE WE COMBINED THE GROUPS BEFORE TESTING THE KNOWN CLASSIFICATION OR BASE RATE IS NOW 50% HALF ADHD AND HALF NONADHD. PPP= RATIO OF ADHD SUBJECTS CONSIDERING ALL WHO TESTED POSITIVE. HIGH PPP MEANS THAT A POSITIVE RESULT RULES IN THE CONDITION NPP= RATIO OF NON/CLINICAL SUBJECTS CONSIDERING ALL WHO TESTED NEGATIVE. HIGH NPP MEANS THAT A NEGATIVE RESULT RULES OUT THE CONDITION * ALONG WITH ACCEPTABLE RELIABILITY & VALIDITY, SOLID SENSITIVITY & SPECIFICITY & PPP & NPP HELP TO ESTABLISH THE EVIDENCE NEEDED TO DETERMINE A TESTS SUITABILITY FOR CLINICAL USE. Slide 13 ADHD POPULATION BASE RATE HOWEVER, THE BASE RATE FOR ADHD IS WELL BELOW THAT OF THE GROUPS IN OUR EXAMPLES ESTIMATES VARY WIDELY BUT ARE OFTEN REPORTED BETWEEN 3 AND 7% FOR OUR WORK AND DEMONSTRATION WE HAVE SELECTED THE CONSERVATIVE ESTIMATE OF 4% LETS SEE HOW MEASURES WITH GOOD SENSITIVITY & SPECIFICITY PERFORM WITH THE CONSERVATIVE BASE RATE Slide 14 EBA CALCULATOR Slide 15 GIVEN THE ADHD BASE RATE OF 4% & VARIED PRESENTATIONS RELIABLE AND VALID ASSESSMENT REQUIRES: 1.MULTIPLE INPUTS 2. EACH WITH ADEQUATE PSYCHOMETRICS 3.DEMONSTRATED DIAGNOSTIC UTILITY EX: ACCEPTABLE PPP&NPP 4.MUST HAVE CLINICAL AND CONTROL GROUPS Slide 16 INCREMENTAL VALIDITY TENETS: MULTIPLE SOURCES OF EVIDENCE USED IN ORDER TO IMPROVE DIAGNOSTIC ACCURACY THE MULTIPLE INPUTS MUST BE JUSTIFIED IN THAT EACH PROVIDES ADDITIONAL NON-REDUNDANT INFORMATION Slide 17 EBA & LIKELIHOOD RATIOS 1. ALLOWS INCREMENTAL INPUTS 2. CAN PROVIDE EVIDENCE FOR OR AGAINST DX 3. CAN CONSIDER EVIDENCE RELATIVE TO THE KNOWN BASE RATE ALLOWS YOU TO CONSIDER THE RELATIVE PREDICTIVE POWER OF A TESTS INDIVIDUAL SCORE POINTS NOT JUST THE OVERALL STATED PERFORMANCE. 4. USES SCIENCE & CLINICAL EXPERIENCE Slide 18 EVIDENCE BASED ASSESSMENT HAS BEEN ADVOCATED SINCE MID1990S & INVOLVES SCRUTINIZING EVIDENCE FOR: SOUNDNESS POWER OF INFERENCE DIAGNOSTIC UTILITY DEVELOPING AN ATTITUDE OF ENLIGHTENED SKEPTICISM TOWARD DIAGNOSTIC PRACTICES Slide 19 LIKELIHOOD RATIOS FEW TESTS ARE ACCURATE ENOUGH TO RULE IN OR OUT DIAGNOSIS ALONE. BEST APPROACH IS TO LOOK AT A GIVEN TEST RESULT AS ALTERING THE PROBABILITY OF AN EXISTENT CONDITION. REQUIRES THE ESTIMATION OF A PRE-TEST PROBABILITY (BASE RATE) Slide 20 LR CONTINUED THE PRE-TEST BASE RATE WILL THEN BE ADJUSTED UP OR DOWN BY THE INPUT OF EACH MEASURE/TEST RESULT ALSO REFERRED TO AS APPLICATION OF BAYESIAN LOGIC. PRODUCES AN ADJUSTMENT FACTOR 1 FOR A RANGE OF PROBABILITY FROM 0 TO 99% Slide 21 EBA CALCULATOR Slide 22 LR CONTINUED CONVERTING TEST SCORE RESULTS INTO LIKELIHOOD RATIOS HELPS DETERMINE HOW USEFUL A DIAGNOSTIC TEST IS HELPS IN SELECTING A SERIES OR SEQUENCE OF TESTS CONSIDERS THE RESULTS IN LIGHT OF THE KNOWN BASE RATE ALLOWS ADDITIVE AND SUBTRACTIVE INPUT TOWARDS THE PREDICTIVE INDEX/OUTCOME Slide 23 STRATEGIES FOR DEVELOPING LIKELIHOOD RATIOS FRAZIER, T.W. & YOUNGSTROM (2006) EVIDENCED-BASED ASSESSMENT OF ATTENTION-DEFICIT HYPERACTIVITY DISORDER: USING MULTIPLE SOURCES OF INFORMATION. JOURNAL AM. ACAD. CHILD ADOLESCENT PSYCHIATRY, 45:5 MAY (2006) Slide 24 STRATEGIES CONTINUED Web resources for EBA: (http://www.childrensmercy.org/stats/categ ory/DiagnosticTesting.asp), &http://www.childrensmercy.org/stats/categ ory/DiagnosticTesting.asp (Centre for Evidence-based Medicine (nd). Likelihood Ratios. Oxford-Centre for Evidence-based Medicine, http://www.cebm.net/likelihood_ratios.asp Slide 25 BASIC STRATEGIES BASICS: LITERATURE REVIEW LOOKING FOR SENSITIVITIES & SPECIFICITIES PUBLISHED FOR GIVEN MEASURES REVIEW CLINICAL MANUALS: EXAMPLES: CDI, RCMAS, BRIEF, COLOR TRAILS, MANY OTHERS LOOK FOR TWO GROUPS CLINICAL AND CONTROLS Slide 26 BASICS CONTINUED DETERMINE SENSITIVITY= % TESTING POSITIVE FROM THE CLINICAL GROUP. FIND THE RELATIVE PERCENTILE OF A GIVEN SCORE POINT(RAW SCORE OR STANDARD SCORE) TO DETERMINE WHAT PERCENTAGE OF THE CLINICAL GROUP FALLS AT THAT GIVEN SCORE POINT. FIND THE NEXT LOWEST RAW SCORE (SCORE POINT) AND SUBTRACT THE CORRESPONDING PERCENTILE FROM 100% EX: RS=20 T-SCORE= 50 %= 50 TH FIND THE PERCENTILE FOR RAW SORE OF 19 AND SUBTRACT THAT CORRESPONDING PERCENTILE FROM 100%. RS 19=% 45 (100-45= 55) SENSITIVITY OF RS 20 =.55 Slide 27 BASICS CONTINUED SPECIFICITY= THE PERCENTAGE OF SUBJECTS WITH A NEGATIVE TEST RESULT FROM THE NON-CLINICAL GROUP. WHAT PERCENTILE OF THE NON- CLINICAL GROUP FALLS AT A GIVEN RAW SCORE OR SCORE POINT FIND THE NEXT LOWEST RAW SCORE AND SUBTRACT THE CORRESPONDING PERCENTILE FROM 100% EX: RS=20 T-SCORE= 50 %= 50 TH FIND THE PERCENTILE FOR RAW SORE OF 19 AND SUBTRACT THAT CORRESPONDING PERCENTILE FROM 100%. RS 19=% 45 (100-45= 55) SENSITIVITY OF RS 20 =.55 IN THIS EXAMPLE THE SENSITIVITY AND SPECIFICITY ARE EQUAL AT.55 TO CALCULATE A CORRESPONDING POSITIVE LIKELIHOOD RATIO USE THE FOLLOWING FORMULA SENSITIVITY/(1-SPECIFICITY).55/(1-.55).55/45 = LR 1.22 Slide 28 DEVELOPING LRS FROM RESEARCH DATA Group a : # of subjects with ADHD, and a positive Test Score. Group b : # of subjects without ADHD, and a positive Test Score. Group c : # of subjects with ADHD, and a negative Test Score. Group d : # of subjects without ADHD, and a negative Test Score. Slide 29 DEVELOPING LRS CONT Sensitivity is the proportion of patients with ADHD who have a positive test. Sensitivity = a / (a + c) Specificity is the proportion of patients without ADHD who have a negative test. Specificity = d / (b + d) Calculate the Ratios: Likelihood ratio (LR+) = sensitivity/(1-specificity) = (a/(a+c))/(b/(b+d)) Likelihood ratio (LR-) = (1-sensitivity)/specificity = (c/(a+c))/(d/(b+d)). The reference information provided above was adapted from the following Web resources for EBA: (http://www.childrensmercy.org/stats/category/DiagnosticTesting.asp), & (Centre for Evidence-based Medicine (nd). Likelihood Ratios. Oxford-Centre for Evidence-based Medicine, http://www.cebm.net/likelihood_ratios.asp Slide 30 PEDIATRIC ADD SCREENING SYSTEM: (SUMMARY OF THE PADDS INPUTS) COMPUTER ASSISTED INTERVIEW ASSESSMENT OF PARENT AND TEACHER DSM-IV RATINGS FOR ADHD COMPLETION OF THE TARGET TESTS OF EXECUTIVE FUNCTIONING COMBINED INPUTS TO ESTABLISH A PROBABILITY INDEX REVIEW OF COMORBIDITY Slide 31 CADI MEDICAL HISTORY/SYSTEMS REVIEW DEVELOPMENTAL HISTORY SOCIAL EMOTIONAL FUNCTIONING DEPRESSION/ANXIETY ATTENTION/HYPERACTIVITY BEHAVIOR/SCHOOL HISTORY Slide 32 Computer Administered/Scored Diagnostic Interview (CADI) EFFECTIVELY ASSESSES FOR COMORBIDITY ESTABLISHES A PRELIMINARY TREATMENT PLAN CAN PROVIDE DOCUMENTATION TO SUPPORT REFERRALS AND OTHER TESTING REQUESTS Slide 33 PEDIATRIC ADD SCREENING SYSTEM Target Tests of Executive Functioning (TTEF) ASSESSES EXECUTIVE FUNCTIONS COMPARES TO ADHD & TYPICAL PEERS CAN EFFECTIVELY RULE IN & OUT ADHD EFFECTIVELY CROSS VALIDATES BEHAVIOR RATINGS Slide 34 Behavioral Inhibition Inhibit Prepotent response Stop an ongoing response Interference control Working Memory Holding events in mind Manipulating or acting on the events Initiation of complex behavior sequences Retrospective function (hindsight) Prospective function (foresight) Anticipatory set Sense of Time Cross-temporal organization of behavior Self-regulation of affect/motivation/arousal Emotional self-control Objectivity / social perspective taking Self regulation of drive and motivation Regulation of arousal in the service of Goal directed action Internalization of speech Description and reflection Rule-governed behavior (instruction) Problem solving / self-questioning Generation of rules and meta-rules Moral reasoning Reconstitution Analysis and synthesis of behavior Verbal fluency / behavioral fluency Goal directed behavioral creativity Behavioral simulations Syntax of behavior Motor control / fluency / syntax Inhibiting task irrelevant responses Excluding goal directed responses Execution of novel / complex motor sequences Goal directed persistence Sensitivity to response feedback Task re-engagement following disruption Control of behavior by internally Represented information Barkleys Model of Behavioral Inhibition Used with permission 1/18/2008 Slide 35 Slide 36 Slide 37 Slide 38 WHY ADD OBJECTIVE MEASURES WHY NOT JUST USE RATING SCALES: SUBJECTIVITY/CAN BE SKEWED DEMAND CHARACTERISTICS MAY ENHANCE PERSONAL LEANINGS CAN BE INCONSISTENT EVEN BETWEEN PARENTS AND MULTIPLE TEACHERS MULTIPLE RATINGS MAY BE CONSIDERED AS REDUNDANT INFORMATION IN SOME CASES BIAS MAY REFLECT MORE ABOUT THE RELATIONSHIP OF THE CHILD & RATER THAN ABOUT THE ORGANIC FUNCTIONING OF THE CHILD. Slide 39 WHY ADD OBJECTIVE MEASURES IF OBJECTIVE MEASURES CAN DEMONSTRATE ACCEPTABLE PSYCHOMETRICS (RELIABILITY & VALIDITY), ALONG WITH ACCEPTABLE PPP & NPP,(DIAGNOSTIC UTILITY) THEY COULD SERVE TO ADD INCREMENTAL EVIDENCE (NON- REDUNDANT) FOR OR AGAINST DX. CAN SERVE TO CROSS VALIDATE THE RESULTS FROM BEHAVIORAL RATINGS ALLOWS FOR BEHAVIORAL OBSERVATIONS CAN HELP LOOK AT EFFECTS OF TREATMENT OR INTERVENTION PARENT AND PHYSICIANS MAY MORE READILY ACCEPT RESULTS IF THEY REALIZE THAT MULTIPLE LINES OF EVIDENCED WERE CONSIDERED IN THE ASSESSMENT/DIAGNOSTIC PROCESS. OBJECTIVE MEASURES FIT WELL INTO A MULTI-METHOD OR EBA APPROACH. Slide 40 Psychometric Properties of the Pediatric ADHD Screener (PADDS) Vann B. Scott, Jr., Ph.D. Armstrong Atlantic State University Savannah, Georgia Slide 41 Clinical sample 629 (265 females & 364 males) children ages 6 to 12 years (M = 8.66, SD = 1.71) Data obtained from 10 sites in 7 states GA35% ID35% TN10% CA10% FL5% NJ3% IL2% Slide 42 Means, standard deviations, standard errors, standard error of measurement, & cut points of three subtests by population Typical vs. Clinical TRTSTTTRTSTT M127.8232.8612.8781.0619.345.91 SD22.286.573.9335.829.623.94 SE1.29.38.251.97.53.24 SEM8.632.541.5213.873.721.52 95% CI 111 - 14528 - 3810 - 1654 - 10812 - 273 - 9 Cut Score > 114> 27> 8 114 27 8 Typical and Clinical participants differed significantly on each of the three subtests, all ts > 19, p