1 development of valid and reliable case studies for teaching, diagnostic reasoning, and other...
TRANSCRIPT
1
Development of Valid and Reliable Case Studies for
Teaching, Diagnostic Reasoning, and Other
Purposes
Margaret Lunney, RN, PhDProfessor
College of Staten Island, The City University of New York
2
What Are Case Studies?Simulations of patients’ stories:
Written, Video-taped, Computer-based
Designed for specific purpose(s)
Variety of designs:Length
Content
Style: Linear, branching
3
When Are Case Studies Used?
Practicee.g., orientation of new nurses, discussion of complex cases
Educatione.g., teaching aids, testing
Researche.g., measurement of accuracy, evaluation of knowledge
4
Advantages: Simulations of Patients’ Stories
Standardized
Represent important, usual, familiar, & challenging situations
Restricts the complexity of practice
User “gets involved”
Reasonable cost
5
Why Is It Difficult to Develop Good Case Studies?
Case studies are tools
Quality= Validity & Reliability
Development takes time, energy and commitment
6
Case Studies as Tools Goal
Strengthen the link between knowledge and application
HowOperationalize complex abstract concepts and “real” patients’ stories, e.g.,
Hope
Caregiver Stress
Ineffective Self Health Management
7
Case Studies as Tools to Measure Nursing Concepts
Nursing Concepts nursing diagnoses
Capture key elements of a real patient situation
Use principles of measurement
Instrumentation-Method to measure concepts
8
Case Studies as Tools (cont.)
Measurement: DefinitionSee Waltz, Strickland & Lenz, 2005
Importance of conceptual frameworksChallenges of measurement
9
Case Studies as Tools (cont.)
The dilemma & challenge: Patients’ stories are complexOverlapping variables Biases in interpretationUse of heuristicsMany types of thinking
GoalReduce ambiguous, abstract ideas to concrete behavioral indicators
10
Measurement Frameworks
Norm-referencedEvaluate performance relative to the performance of others
Criterion-referencedEvaluate performance based on
pre-determined
standards
11
Case Studies as Criterion-Referenced Tools: Reliability
Definition:Consistency of scoring
Range of variability is reduced, use nonparametric procedures
Test-Retest
Parallel forms
Interrater & intrarater agreement
12
Case studies as Criterion-Referenced Tools: Validity
Definition:Measures what was intended, systematic error is reduced
Types:Content validity
Criterion-related validity
Construct validity
13
Case Studies as Criterion-Referenced Tools: Tasks
Precisely specify target behaviors Identify standards for target
behavior Discriminate who did and did not
acquire the target behavior Compare subjects’ performance to the standards
14
Case Studies as Tools (cont.)
Types of written simulationsLinear
BranchingFree branching
Modified free branching
Forced branching
15
Linear Technique
Subjects follow same sequenceOne or more sectionsIf more than one section, students receive specific instructions, and
Sections are appropriate for all subjectsOne section does not influence other sectionsEach section samples inquiries or actions
16
Guidelines for Case Study Development: Linear Format1. Identify the overall purposes:
What is the general topic?What kind of problem solving will be represented?How important are the responses, e.g., learning process, grade, research data?
17
Guidelines (cont.)
2. Specify objectives, e.g., toMeasure accuracy of diagnosis
Illustrate relation of cues to inferences
Facilitate planning
Orient new nurses
Teach sequential decision making
18
Guidelines (cont.)
3. Decide the complexity: Length
Number of diagnoses, interventions,
and/or outcomes
Amounts of high, moderate & low
relevance data
Types of data
19
Guidelines (cont.)
4. Obtain literature sources & nurse experts to support validity:
Select authoritative sources: actual cases, literature sources Decide importance of each sourceIdentify expert judges, e.g., may ask colleagues Use well-known experts for research tools
20
Guidelines (cont.)
5. Formulate case studies, directions, & scoring manual
Directions must be explicit,
comprehensive, and clearly written
Scoring manual provides scores for possible answers
21
A) Create Blueprint
Choose a general problem areaWhat should this exercise teach or test?
What kinds of problems need to be solved?
What kinds of knowledge needs to be used?
Define the objectives to be sampled
Select concrete problem situation(s)
Outline and diagram the exercise
22
B) Prepare Specifications
Method of administration
Proportion of content
for each objective
Style of writing case(s)
Restrictions, e.g. time
Describe scoring procedures
23
C) Construct Case Studies
Develop pool of
data/items to
match objectives
Review to determine content
validity and appropriateness
Edit, delete, change as indicated
Assemble the case(s)
24
D) Set Standards or Cut Scores
What types of answers
are “correct”
What answers/scores are
acceptable or unacceptable
Classification of various
answers, # of points toward a
grade
25
Guidelines (cont.)
6. Obtain content validityratings from experts:
Select content validity method Item-objective congruenceInterrater agreementAverage congruency percentage
Send with explicit instructions Be prepared to make changes
26
Guidelines (cont.)
7. Evaluate with a pilot test:Consider demographics, e.g., Experience Knowledge
Evaluate directions, restrictions, scoring methods, interrater reliability