using script concordance testing to assess...
TRANSCRIPT
Using Script Concordance for Clinical Reasoning
Assessment Rebecca Maldonado, MS, PA-C
Associate Professor, University of Colorado Physician Assistant
Program
Kristine Himmerick, MS, PA-C
Assistant Professor, Northern Arizona University Physician
Assistant Program
Session Objectives 1. Describe the theory of script concordance testing
(SCT) as a measure of clinical reasoning.
2. Understand the reliability of script concordance
testing.
3. Recognize the parameters for constructing an
expert clinician panel.
4. Comprehend the development techniques for a
script concordance test item.
5. Understand the use of the SCT calculator for
performing item analyses for both the expert panel
responses and learner responses to SCT items.
What is Clinical Reasoning?
• “the thinking and decision-making processes associated
with professional practice”
o-Higgs
Hierarchy of Clinical Reasoning
• Knowledge o Didactic knowledge
o Clinical skills
• Cognition o Application of knowledge to practice
• Metacognition o Thinking about the decision making process
o Thinking about thinking (Flavell, J. H. 1979)
Awareness of the learning process
Recognition of the limitations
of memory
Appreciation of perspective
Capacity for self-critique
Ability to select appropriate strategies for different
situations
(Croskerry, 2003)
Metacognition
• Foundational knowledge and experience are
required to progress o Novice - Advanced beginner – Competence – Proficiency – Expert
• Beginners are taught rules with little context.
• As the learner gains experience in applying the
knowledge to real world situations, he/she
advances toward expertise. (Benner, 1982)
Novice to Expert Continuum
Novice to Expert Thinking
Novice Monitoring, Reflection
Developing
Experienced
More
Less
Low High
Reliability, Accuracy
Cost, Time,
Effort
(Graber 2009)
Expert
How can medical educators evaluate clinical reasoning?
Can the novice to expert continuum be objectively
measured?
Scripts are… • Medical knowledge structures which link
relevant clinical and pathophysiologic information
• Scripts frame an individual’s expectations (hypotheses) in clinical scenarios
• Initiated during early medical training AND developed through clinical exposure and practice.
Scripts Concordance Testing is… … a written assessment of
clinical reasoning under
conditions of uncertainty
…clinical data interpretation
…quality of acquired scripts
Specific facet of
clinical reasoning
competence:
Hinges on:
A proxy
indicator of: …EXPERTISE
‘Gold Standard’:
Acknowledged experts in field
Script Concordance Test (SCT)
Multiple Choice Questionnaire (MCQ)
Ill-defined problem with context of uncertainty
Well-defined problem with no uncertainty
More than one
acceptable answer
Only one correct answer
Assess Reasoning
Assess Knowledge
Clinical Reasoning: Key Steps
SCT Question Format
Patient history and nonverbal
communication Script
Activation
Case Vignette
Hypothesis Generation
"If you were thinking…"
Data Collection Data Interpretation
"And then you find…"
Hypothesis Evaluation
"Your hypothesis becomes more or less
likely?" (-2, -1, 0, +1, +2)
SCT Principles 1. Examinees are faced with challenging,
realistic clinical situations and must choose between several appropriate response options
2. Item format reflects the way information is processed in clinical problem-solving situations
3. Scoring takes into account the variability of responses of experts to clinical situations
SCT Reliability • Alpha-coefficients of 0.70-.090
across multiple disciplines
(Lubarsky, 2011)
Creating SCT Items
• Create a small case vignette
• Must contain ambiguity or be missing information
• Appropriate for examinee’s training
Example:
Matt is a 26-year old male who presents to the ED
with a history of left-sided chest pain and shortness of
breath.
Creating SCT Items
• Select hypotheses for testing • Must be plausible
• Represent common or life-threatening possibilities
Examples:
• Pneumonia
• Pulmonary embolism
• Acute coronary syndrome
• Spontaneous pneumothorax
• Rib fracture/chest trauma
• Anxiety/panic attack
Creating SCT Items • Write the SCT item
• Test the link of new information into the context of the clinical
vignette
• Introduce key features which may have a broad range of
possible responses among clinicians
• Write items which assess reasoning within the case vignette
across the spectrum of the response scale
Example:
If you were thinking: And then you find: This diagnosis becomes:
Pneumonia The patient is afebrile -2 -1 0 1 2
Spontaneous Pneumothorax
The onset of chest pain/SOB was while running
-2 -1 0 1 2
Acute coronary syndrome
The patient uses cocaine -2 -1 0 1 2
-2: Very unlikely , -1: Somewhat unlikely, 0: Neither more or less likely, 1: Somewhat likely, 2: Very likely
Creating Investigative/Therapeutic SCT Items
• Write the SCT item
• Test the link of new information into the context of the clinical
vignette
• Introduce key features which may have a broad range of
possible responses among clinicians
• Write items which assess reasoning within the case vignette
across the spectrum of the response scale
Example:
If you were thinking of: And then you find: This treatment becomes:
Ordering a chest x-ray The patient is not tachypneic -2 -1 0 1 2
Ordering troponin levels
The patient’s EKG is normal -2 -1 0 1 2
Ordering a D -dimer The patient’s Well’s criteria score was <2
-2 -1 0 1 2
-2: Contraindicated , -1: Less indicated 0: Neither more or less indicated, 1: Somewhat indicated, 2: Very indicated
Creating SCT Items: More Cases or More Questions?
Gagnon et al., 2008
Vetting the SCT items: 1. Construct an Expert Panel
A. Utilize faculty or community providers with
experience in item content areas(Petrucci, 2013)
B. Expert panel size 10-15 members (Gagnon, 2005)
0,40
0,45
0,50
0,55
0,60
0,65
0,70
0,75
0,80
n=5 n=10 n=15 n=20 n=25 n=30 n=38
mean (sd) min value max value
Vetting the SCT items: 2. Scoring Expert Panel Responses
A. Discard or re-write items identified by panel as confusing,
erroneous, etc.
B. Use aggregate responses, don’t look for consensus (Norman,1985)
C. Score the responses using modal distribution (Charlin, 2010)
D. In panels <15, remove “outlier” responses (Gagnon, 2011)
Scoring Key for 1 SCT item (panel size= 10)
Answers -2 -1 0 1 2 (1) (2) (3) (4) (5)
Number of panel members Who chose this answer 0 0 2 5 3
Number of panel members Who chose this answer divided By the modal answer 0/5 0/5 2/5 5/5 3/5
Score for each response On this item 0 0 .4 1 .6
Vetting the SCT items: Choosing Moderate Variability Items
Use SCT calculator to determine variability of panel
responses for each item.
Staff-students
0
0,5
1
1,5
Low variability Moderate variability High variability
Charlin, 2006
SCT Calculator: Expert Panel Response Entry
1. Must be entered using 5-point Likert numbering 2. Once expert panel responses are entered, click to the 3rd worksheet tab to see item statistics
SCT Calculator: Expert Panel Item Stats
http://www.cpass.umontreal.ca/sct.html [email protected] for Excel file
A.Very likely
B. Somewhat more
likely
C.Neither more or
less likely
D.Somewhat
unlikely
E. Very unlikely
SCT Item #1 - Audience Response:
If you were thinking: And then you find:
PleuritisChest pain began
with Exercise
A)
Very
likely
B)
Somewhat
more
likely
C)
Neither
more or
less likely
D)
Somewhat
unlikely
E)
Very
unlikely
This diagnosis becomes:
A. B. C. D. E.
20% 20% 20%20%20%
Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath.
A.Very likely
B. Somewhat more
likely
C.Neither more or
less likely
D.Somewhat
unlikely
E. Very unlikely
SCT Item #2 - Audience Response:
A. B. C. D. E.
20% 20% 20%20%20%
Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath.
If you were thinking: And then you find:
Chest trauma / pain /
strain
Chest pain was
improved by lying
down
A)
Very
likely
B)
Somewhat
more
likely
C)
Neither
more or
less likely
D)
Somewhat
unlikely
E)
Very
unlikely
This diagnosis becomes:
A.Very likely
B. Somewhat more
likely
C.Neither more or
less likely
D.Somewhat
unlikely
E. Very unlikely
SCT Item #3 - Audience Response:
A. B. C. D. E.
20% 20% 20%20%20%
Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath.
If you were thinking: And then you find:
Pneumothorax
Patient exam
revealed a mid-line
trachea
A)
Very
likely
B)
Somewhat
more
likely
C)
Neither
more or
less likely
D)
Somewhat
unlikely
E)
Very
unlikely
This diagnosis becomes:
Expert Panel Responses: SCT Items 1-3
Administering the SCT Items
1. Options for student exams
2. SCT calculator for student scoring & item
analysis
SCT Calculator- Scores &Item
Analysis of Learner Responses
Avoiding Pitfalls: Use this “Do” list
1. Write short clinical vignettes
2. Write questions for which the answer is not clearly
defined in books or clinical guidelines.
3. Keep the number of questions per case to < 5.
4. Solicit more expert panel members than you need by
at least 30%.
Questions
Many thanks to Kevin Bogart, MS, PA-C for his contributions to this presentation.
References Benner, P. (1982). “From novice to expert”. American Journal of Nursing, 82(3), 402-407.
Charlin, B. (2010). “Assessment in the context of uncertainty using the Script Concordance test: more meaning for scores”. Teaching and Learning in Medicine, 22(30), 180-186.
Charlin B, Boshuizen H, Custer E, Feltovich P. (2007 )”Scripts and clinical reasoning”. Medical Education. 41:1178-84.
Charlin B, Gagnon R, Pelletier J, et al. (2006 )”Assessment in the context of uncertainty: the effect of variability within the panel of reference”. Medical Education.18: 22-7.
Croskerry, P. (2003). "Cognitive forcing strategies in clinical decisionmaking." Ann Emerg Med 41(1): 110-120.
Flavell, J.H. (1979). “Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry”. American Psychologist, 34(10), 906-911.
Fournier J, Demeester A, Charlin B. (2008 )”Script Concordance Tests: Guidelines for construction”. BioMedCentral, Medical Informatics and Decision Making. 8: 18.
Gagnon, R, et al. (2008) “Script Concordance testing: more cases or more questions”. Adv. Health Sci Educ. 14: 367-375.
Gagnon R, Charlin B, Coletti M, Sauve E, van der Vleuten C. (2005) “Assessment in the context of uncertainty: How many members are needed on the panel of reference of a script concordance test”. Medical Education. 39: 284-91.
Graber, M. L. (2009). "Educational strategies to reduce diagnostic error: can you teach this stuff?" Adv Health Sci Educ Theory Pract 14 Suppl 1: 63-69.
Higgs, J. and M. A. Jones (2000). Clinical reasoning in the health professions. Oxford ; Boston, Butterworth-Heinemann.
Lubarsky S, Charlin B, Cook D, Chalk C, van der Vleuten C. (2011 ) “Script concordance testing: a review of published validity evidence”. Medical Education. 45:329-338.
Norman, G. (2005). Research in clinical reasoning: past history and current trends. Med Educ, 39(4), 418-427.
Petrucci, AM, et al. (2013) “Assessing clinical judgment using the Script Concordance test: the importance of using specialty-specific experts to develop the scoring key”. Am J Surg. 205(2): 137-140.