chapter 13 evaluation and diagnosis: research methods and data analysis

Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins

Chapter 13Evaluation and Diagnosis: Research Methods and Data

Analysis

Chapter 13Evaluation and Diagnosis: Research Methods and Data

Analysis


Chapter OverviewChapter Overview• Health care providers use a variety of diagnostic procedures, including

laboratory testing, diagnostic imaging, physical examination, interviewing, and observation.

• Patient interviews and the performance of physical examination procedures are the primary tools available to those evaluating an individual seeking care.

• Overview of the diagnostic continuum and the differences between research into diagnostic testing and prevention or treatment strategies.

• Define and decrease investigational bias.

• Concepts of sensitivity and specificity.

• Calculation of sensitivity and specificity values, and likelihood ratios.

• Positive and negative prediction values.

• Understand and interpret receiver-operator characteristic (ROC) curves.


Physical Examination Procedures Physical Examination Procedures • Observation, interview, and the performance of physical

examination procedures are the primary methods available during the injury evaluation process.

• Observation guides the opening questions in an interview with a patient. Through observation and interview, the list of diagnostic possibilities shrinks, and one or two conditions emerge as the most likely culprits of the patient’s complaints.

• The clinician can then select physical examination procedures that will help confirm or refute the existence of the suspect conditions.

• Research into the performance of diagnostic tests differs from research into, for example, prevention efforts and treatment outcomes.

• In studies of diagnostic procedures, comparison to another group of patients or subjects is unnecessary. These studies require that the results of the diagnostic test of interest be compared to the results of an established standard.


Clinical Research Improving Patient EvaluationClinical Research Improving Patient Evaluation

• Challenge in health care: maximize benefit to the patient without squandering talent, time, and money on procedures that are of little benefit or that pose more risk than benefit.

• The only way to find answers to clinical questions is through clinical trials.

• The true magnitude of benefit and risk are revealed once a diagnostic test is studied in the population it is intended to benefit.

• The clinician-consumer of clinical research must understand the relationship between study methods and threats to the validity of data.

• The methods of studies of prevention, treatment, and diagnostic procedures, and therefore the steps that minimize investigational bias, differ.


Design of Studies of Diagnostic TestingDesign of Studies of Diagnostic Testing

• Estimating the usefulness of any diagnostic procedure involves comparing the results of a diagnostic test to the results of an established, “gold” standard.

• Statistical analysis for studies involving tests with dichotomous outcomes involves positive or negative results.

• If a test is on a continuous scale, then a different analysis is required to assess diagnostic test performance; but a gold standard identifying those with and without the diagnosis of interest is still needed to estimate the usefulness of the test of interest.

• The reason a gold standard test is not applied to all diagnoses in question has to do with the setting, timing, scope, costs, and risks associated with the diagnostic process. These issues often overlap.


Physical Examination Used to Narrow Down Diagnostic PossibilitiesPhysical Examination Used to Narrow Down Diagnostic Possibilities

• Studies of diagnostic procedures guide clinicians in the certainty of their diagnoses.

• The less certain one is of a particular diagnosis, the greater the likelihood that something else is wrong.

• The use of advanced diagnostic procedures (e.g., MRIs) comes at a cost and in some cases (e.g., spinal tap to work up a patient who may have meningitis) with a risk.

• The research consumer must decide if the subjects enrolled in a study of diagnostic tests were sufficiently similar to patients in their care to support the generalization of the results.

• If after considering the generalizability of research findings into one’s practice the results of the paper remain of interest, the research consumer must consider the validity of the data reported.


Assessing Research of Diagnostic InstrumentsAssessing Research of Diagnostic Instruments

• 14-item assessment tool: assists consumers in evaluating the methodological quality of research into diagnostic tests. Scores of 10 or more are considered to reflect sound research methods.

• Subject selection has the potential to bias results as well as to influence the generalization of research findings.

• Studies should include a spectrum of patients to whom the test in question would typically be applied in a clinical setting.

• Spectrum bias: exists when only patients very likely to be suffering from the condition of interest are studied or when patients that clearly do not have a condition are included.

• Blinding of investigators performing or interpreting the results of the diagnostic test being investigated as well as those responsible for the results of “gold standard” assessment to each other’s finding is essential (items 10 and 11).

• “Work-up” bias: data are biased towards underreporting of false negative results.

• Some tests may not yield useful data, and not all subjects can complete all testing (items 11 and 12).


The QUADAS Tool


The QUADAS Tool (continued)The QUADAS Tool (continued)

Reprinted with permission from Whiting P, et al. BMC Med Res Meth. 2003;3:25.


Statistics and InterpretationsStatistics and Interpretations• Research reports from diagnostic tests describe how the data were

analyzed and provide estimates of the diagnostic test performance.

• The results from studies of diagnostic tests are reported in several forms:

– estimates of sensitivity and specificity– positive and negative prediction values– positive and negative likelihood ratios– receiver operator characteristic curves

• When one compares the results from a diagnostic test with a dichotomous result (positive or negative) of interest to the results of a “gold standard” diagnostic test, four subgroups or cells are formed.

• From this table sensitivity and specificity, positive and negative prediction values, and positive and negative likelihood ratios can be calculated.

• An expansion of the table vertically is needed to generate receiver operator characteristic curves.


Diagnostic Test with a Dichotomous Result Compared to “Gold Standard” Diagnostic Test Results

Diagnostic Test with a Dichotomous Result Compared to “Gold Standard” Diagnostic Test Results


Sensitivity and SpecificitySensitivity and Specificity• Sensitivity and specificity are related to the ability of a test to identify

those with and without a condition and are needed to calculate likelihood ratios.

• Sensitivity is the number of illnesses or injuries that are correctly diagnosed by the clinical examination procedure being investigated (cell A) divided by the true number of illnesses/injuries (cells A + C) (gold standard measure).

# diagnosed as having a condition by clinical examination procedure =_A_# diagnosed as having a condition based upon gold standard A+C

• Specificity is the number of individuals correctly classified as not having the condition of concern based on the test being investigated (cell D) divided by the true number of negative cases (cells B + D) (gold standard measure). *Calculated from the right side of the table.*

# diagnosed as not having a condition by clinical examination procedure = _D_ # diagnosed as not having a condition by gold standard B+D


Impact of Sensitivity andSpecificity on Clinical PracticeImpact of Sensitivity andSpecificity on Clinical Practice

• Ideally, a diagnostic procedure has high sensitivity and high specificity. Unfortunately many diagnostic tests lack sensitivity or specificity or both.

• SNout: tests with high sensitivity are good at ruling out a condition

• SPin: tests with high specificity are good at ruling in a condition

• A sensitive test is one with relatively few false negative findings; thus, a negative examination finding effectively rules out the condition of interest.

• Tests with high specificity have few false positives; thus, a positive result using a diagnostic test with high specificity has identified the target disorder.


Positive and Negative Prediction ValuesPositive and Negative Prediction Values• The calculation and interpretation of sensitivity and specificity are predicated

on the effort to identify individuals with and without a target disorder of interest.

• These values are calculated “vertically” using cells A and C or B and D respectively:

• Positive prediction value: if a diagnostic test is positive, what is the probability the target condition is present? Calculated by: PPV= A / A+B

• Negative prediction value: estimates that the target condition is not present when the diagnostic test is negative. Calculated by: NPV = D / D+C

• Sensitivity and specificity are calculated differently than PPV and NPV.

• As prevalence falls, PPV values also fall and NPV values rise.

• Sensitivity and specificity values are more stable estimates than PPV and NPV but are difficult to apply in clinical practice.


Likelihood RatiosLikelihood Ratios• Likelihood ratios (LRs): values derived from estimates of sensitivity and

specificity.

• LRs can be applied by the clinician to the examination of individual patients.

• Knowledge of the LRs influences the level of certainty that a condition does or does not exist at the end of the examination.

• A positive likelihood (+LR) ratio: impact of a positive examination finding on the probability that the target condition exists. For tests with dichotomous results, an +LR is calculated as follows:

– (+) LR = Sensitivity / (1 – Specificity)

• A negative likelihood (-LR) ratio: impact of a negative examination on the probability that the condition in question is present. Negative likelihood is calculated as follows:

– (–) LR = (1 – Sensitivity) / Specificity

• A negative result of a diagnostic test with a small likelihood ratio suggests that the chance that the target condition exists is very low.


Likelihood Ratios: Broader Categories of Clinical ValueLikelihood Ratios: Broader Categories of Clinical Value

• Jaeschke et al. (1994) summarized likelihood ratios (positive and negative) into broader categories of clinical value, as follows:


Applications Applications • Diagnosis is a process during which information is gathered to narrow the range

of diagnostic possibilities.

• A carefully conducted interview and observation form the foundation of a physical examination.

• As the examination proceeds, a narrowing list of diagnostic possibilities is developed.

• Once the diagnostic possibilities have been narrowed, the physical examination and other diagnostic assessments that may confirm or rule out specific diagnoses begins.

• A level of suspicion regarding diagnostic possibilities exists before the examination procedure is performed.

• This pretest level of suspicion can be quantified as pretest probability.

• Pretest probability values vary among clinicians and the circumstances of the individual patient.


Probability and UncertaintyProbability and Uncertainty• Probability implies uncertainty. Uncertainty is inherent in the clinical practice

of athletic training.

• Decisions regarding referral, plans of treatment, and a physician’s use of additional diagnostic studies revolve around the level of certainty (probability) that a condition does or does not exist.

• Probability can be converted into an odds ratio using the following formula:Odds = probability / (1 – probability)

• Posttest probability is calculated by first multiplying the pretest odds by the LR+ to yield posttest odds.

• The conversion to posttest probability is made using the following formula:Formula posttest probability = posttest odds / (posttest odds + 1)

• “Fair degree” of uncertainty: the more serious the consequences of being wrong, the broader the definition of uncertainty. Additional testing is indicated when this exists.


Receiver-Operator Characteristic CurvesReceiver-Operator Characteristic Curves• Applications in data management:

– The assessment of diagnostic procedures in which data from a procedure of interest are on a continuum rather than dichotomous (positive or negative).

• ROCcs are an extension of the sensitivity, specificity, and likelihood ratios.

• ROCcs allow for the identification of critical points along continuous measures to guide clinical practice.

• ROCcs can also be applied to the analysis of clusters of diagnostic or prognostic criteria.


Chapter Summary and Key Points Chapter Summary and Key Points • Diagnostic testing should be used as an aid instead of a crutch for a clinician.

• History and observation should help narrow the scope of possible injuries during the evaluation process.

• Diagnostic tests have limitations.

• The only way to find answers to clinical questions is through clinical trials.

• Special tests do not always deliver clear-cut decisions.

• The amount of confidence a clinician has in assessment results determines the likelihood of a correct diagnosis.

• Clinical research must be sorted through because of the varying results in the data.

• Comparing the clinical study with the current standard is important in research.

• Statistics are crucial to research methods because they provide numerical evidence that can be ranked to form valid conclusions.

chapter 13 evaluation and diagnosis: research methods and data analysis

Documents