radiology residents’ on-call interpretation of chest radiographs for pneumonia1

Radiology Residents’ On-call Interpretation ofChest Radiographs for Pneumonia1

Oreoluwa Ojutiku, MD, Linda B. Haramati, MD, Saul Rakoff, MD, Seymour Sprayregen, MD

Rationale and Objectives. This study is designed to assess the performance of radiology residents in interpreting emer-gency department chest radiographs for pneumonia and to characterize chest radiographic findings in patients for whichinterpretation was amended by an attending radiologist.

Materials and Methods. We retrospectively reviewed all amended reports for chest radiographs performed on emergencydepartment patients July 2002–June 2003. Reports preliminarily interpreted by residents and amended by a board-certifiedstaff radiologist for the presence or absence of pneumonia were identified. A panel of three experienced radiologists,blinded to reports, jointly reviewed each chest radiograph. If the panel diagnosed pneumonia, the chest radiograph wasevaluated for the projection that best showed the pneumonia, its size and location, and the presence or absence of the fol-lowing features: increased opacity, air bronchograms, loss of vascular markings, silhouette sign, and linear opacities. Theresident’s post-graduate year (PGY) training level was noted.

Results. One percent (134/12,600 reports) of chest radiographic reports were amended for the presence or absence of pneumo-nia. One hundred chest radiographs were available and comprised the series. There were 56 females and 44 males with a meanage of 45 years (range, 1–99 years). The staff radiologist diagnosed pneumonia in 79% (79/100 radiographs). The panel agreedwith the staff in 77% (� � 0.76) and the resident in 23% (� � 0.43). The panel diagnosed pneumonia in 60% (60/100 radio-graphs) with the following chest radiographic findings: 100% (60/60), increased opacity; 37% (22/60), air bronchograms; 72%(43/60), loss of vascular markings; 40% (24/60), silhouette sign; and 20% (12/60), linear opacities. The pneumonia was rightsided in 52% (31/60), left sided in 37% (22/60), and bilateral in 11% (7/60). Right-sided pneumonias were equally distributedamong the three lobes, and left-sided pneumonias had a lower-lobe predominance of 77% (17/22). Seventy-five percent (45/60)of pneumonias were segmental or smaller, and 82% (49/60) of chest radiographs showing pneumonia had both posteroanteriorand lateral projections. The pneumonia was conspicuous on only one projection in 43% (21/49); the posteroanterior view in22% (11/49), and the lateral view in 20% (10/49). Eighty-one percent (81/100) of interpreting residents were PGY-3.

Conclusion. Interpretation of chest radiographs for pneumonia by PGY-3 residents has a low error rate. Missed pneumo-nias often were segmental or smaller and conspicuous on only one projection.

Key Words. Radiography; chest; pneumonia; resident radiology.© AUR, 2005

Pneumonia is the seventh leading cause of death and theleading cause of death from infection in the United States(1), resulting in more than four million hospitalizations

Acad Radiol 2005; 12:658–664

1 From the Department of Radiology, Albert Einstein College of Medicine,Montefiore Medical Center, 111 East 210 Street, Bronx, NY 10467. Re-ceived June 8, 2004; Accepted June 12. Address correspondence to:L.B.H. e-mail: [email protected]

©
AUR, 2005doi:10.1016/j.acra.2004.06.010
658

annually (2). The diagnosis of pneumonia relies heavilyon a high index of clinical suspicion, with several symp-toms that are associated strongly with the diagnosis, in-cluding fever in up to 80% and respiratory rate greaterthan 20 breaths/min and crackles on auscultation in up to80% of patients (3). The chest radiograph is the tool ofchoice to confirm the diagnosis (4).

Chest radiographic diagnosis of pneumonia usually isstraightforward. Typical findings include increased opac-
ity in a lobar, sublobar, or patchy distribution, often asso-

Academic Radiology, Vol 12, No 5, May 2005 RADIOLOGY RESIDENTS AND CHEST RADIOGRAPHS

ciated with air bronchograms. The “silhouette sign” canbe helpful in confirming the presence of and localizing aparenchymal abnormality. Ancillary findings, such aspleural effusion, also may be present (4).

At many centers, radiology residents and fellows ofteninitially interpret radiographic and other imaging studiesduring off-hours. The images subsequently are reviewedby a staff radiologist. If there is a clinically significantdiscrepancy between the preliminary and final interpreta-tion, an amended report is issued.

The initiation of treatment for pneumonia often is dic-tated by clinical impression and interpretation of the chestradiograph. Therefore, an amended chest radiographicreport diagnosing or refuting the diagnosis of pneumoniacan lead to important changes in patient care. Physicalexamination alone or chest radiograph interpretation bynonradiologists usually is not sufficient to accurately diag-nose pneumonia, especially in atypical clinical presenta-tions (5,6).

Past studies have shown discrepancy rates betweenresident and attending interpretations of computed tomo-graphic (CT) scans (body and brain) and ultrasounds be-tween 1.7% and 9%, with an average discrepancy rate of5% (7–13). Reduced error rates have been describedwhen multiple readings are performed (14–16). Somestudies have characterized the level of agreement betweenvarious groups of interpreters in the diagnosis of pneumo-nia by plain films (17–21). However, there is a paucity ofliterature detailing characteristics of cases that prove diffi-cult to diagnose.

The purpose of the present study is to identify discrep-ancies between the preliminary resident interpretation andfinal staff interpretation of chest radiographs when thediagnosis of pneumonia was in question. In addition, wehighlight radiologic findings commonly associated withdiscrepant interpretations.

MATERIALS AND METHODS

To identify cases of pneumonia that had discrepantinterpretations between the resident’s preliminary reportand the final interpretation, we retrospectively reviewedall amended reports generated for chest radiographs onpatients from the emergency department between July2002 and June 2003. In each case, the chest radiographwas interpreted initially by a radiology resident and anamended report subsequently was generated by a board-
certified staff radiologist. Generation of an amended re-
ported is mandated by our departmental policy when aclinically significant difference exists between the resi-dent’s preliminary interpretation and the final report. Ap-proximately 12,600 chest radiographs from the emergencydepartment were interpreted preliminarily by a radiologyresident during the study period. Key words indicative ofpneumonia were sought in the amended report to facilitatethe identification of study cases. These key words in-cluded pneumonia, infiltrate, consolidation, opacity, airbronchograms, and the silhouette sign.

One percent (134/12,600) of chest radiographs had areport that was amended for the presence or absence ofpneumonia. One hundred of 134 (75%) chest radiographswere available for review and comprised the studypopulation.

Radiology residents who initially interpreted the chestradiographs had a minimum of 9 months of training. Allresidents passed a credentialing examination (22) showingtheir readiness to interpret emergency department imag-ing. Level of resident training was recorded according tothe number of years after graduation from medical school(post-graduate year [PGY]). The vast majority of residenton-call interpretations in our department are performed byPGY-3 and PGY-4 residents, with a 2:1 ratio. The sub-specialty of the staff radiologist amending the resident’sinitial interpretation was recorded.

A panel of three radiologists blinded to all prior re-ports jointly reviewed each chest radiograph. The panelincluded two subspecialized thoracic radiologists and asenior radiologist with extensive experience in chestradiology.

For each chest radiograph, a medical student presentedthe examination to the panel and each radiologist wasasked to decide whether pneumonia was present. The casethen was discussed by the panel and a unanimous or ma-jority interpretation was obtained. Any technical issuesthat made films more difficult to interpret were noted dur-ing the panel’s discussions.

If pneumonia was diagnosed by the panel, the chestradiograph was evaluated for the projection that best de-picted the pneumonia and its size and location. The pres-ence of specific radiographic findings that may indicatepneumonia were sought as follows: increased opacity, airbronchograms, loss of vascular markings, silhouette sign,and linear opacities.

The study included 100 patients. There were 56 fe-males and 44 males with a mean age of 45 years (range,1–99 years). Eighty-seven patients were adults and 13
were pediatric patients younger than 21 years. Posteroan-
659

OJUTIKU ET AL Academic Radiology, Vol 12, No 5, May 2005

terior and lateral views of the chest were performed in85% (85/100), and only an anteroposterior view was per-formed in 15% (15/100).

Agreement rates between the resident’s preliminaryreading and the staff radiologist’s final interpretation werecalculated, with the panel diagnosis considered the “goldstandard.” � Statistics and their 95% confidence intervalswere determined, with � agreements as a measure of theinter-rater reliability that adjusts for random agreement(17). � Values that fall within the following range gener-ally are accepted: 0.0–0.2 � slight agreement, 0.21–0.40� fair agreement, 0.41–0.60 � moderate agreement,0.61–0.80 � substantial agreement, and 0.81–1.00 � per-fect agreement (18).

The institutional review board approved this study.

RESULTS

Interpreter CharacteristicsEighty-one percent of amended reports for the pres-

ence or absence of pneumonia were interpreted initiallyby radiology residents in the PGY-3 level (Table 1).Board-certified staff radiologists had expertise in a varietyof subspecialties. Thirty percent of amended reports forpneumonia were generated by radiologists with expertisein chest radiology, and another 30% were generated byradiologists with expertise in abdominal imaging. Twelvepercent (12/100) of chest radiographs were amended bypediatric radiologists. Of these 12 chest radiographs, 50%(6/12) were in pediatric patients. General, musculoskele-tal, and ultrasound radiologists amended the remaining28% (28/100) of chest radiographs.

Diagnosis Made by the InterpretersSeventy-nine percent (79/100) of amended chest radio-

graphs were interpreted preliminarily as normal by a radi-ology resident and subsequently amended to report pneu-monia (Figure 1). Twenty-one percent (21/100) ofamended chest radiographs were interpreted preliminarilyas pneumonia and amended to report no pneumonia. Themajority interpretation of the panel diagnosed pneumoniain 60% (60/100) of the study population. Of these 60cases, a unanimous interpretation for pneumonia wasachieved in 72% (43/60) and a majority interpretation wasachieved by agreement of two of the three panel membersin 28% (17/60). Among the 40 cases diagnosed by thepanel as no pneumonia (Figure 2), a unanimous interpre-tation was achieved in 88% (35/40), and a majority inter-
pretation, in 12% (5/40).
660

Interobserver AgreementsPanel agreement on the presence or absence of pneu-

monia with the observer groups was as follows: staff radi-ologists, 77%, and radiology residents 23%.

Table 2 lists weighted � agreement values. Greateragreement occurred between the attending physicians andthe expert panel (� agreement, 0.76) compared with thatobserved between the expert panel and residents (� agree-ment, 0.428).

Characteristics of Pneumonia Cases Diagnosedby the Expert Panel

Eighty-two percent (49/60) had both posteroanteriorand lateral projections. The pneumonia was conspicuousto the panel on only one projection in 43% (21/49); pos-teroanterior only in 22% (11/49), and lateral only in 20%(10/49). Pneumonia was seen well on both projections in57% (28/49).

Of 60 cases of pneumonia, 75% (45/60) were segmen-tal or smaller (Figure 1), 20% (12/60) were multisegmen-tal, and 5% (3/60) were lobar in distribution.

Chest radiographic findings of pneumonia were rightsided in 52% (31/60), left sided in 37% (22/60), and bi-lateral in 11% (7/60). Right-sided pneumonias wereequally distributed among the three lobes, whereas left-sided pneumonias had a lower-lobe predominance of 77%(17/22). Fifty-seven percent (4/7) of bilateral pneumoniaswere bibasilar.

Specific findings characterizing these cases includeincreased opacity (100%), loss of vascular markings(72%), presence of the silhouette sign (40%), air bron-chograms (37%), linear opacity (20%), and nodular opac-ity (18%). Lymphadenopathy was present in only 2% ofcases (Table 3).

DISCUSSION

Discrepancies between initial resident and final attend-

Table 1PGY of Radiology Residents Who Interpreted the 100 CasesAmended for the Presence or Absence of Pneumonia

PGY Level Cases (%)

2 63 814 125 1

ing physician interpretations have been described for vari-

air b


ous imaging modalities. Carney et al. (7) compared pre-liminary and final interpretations of after-hours CT scansand sonograms by residents and attending physicians andfound a discrepancy rate of 3.8%. The majority of errorswere minor and did not impact on patient outcome. Adiscrepancy rate of 7.7% was described by Wechsler etal. (8) between the initial trainee and final attending inter-pretations of emergency body CT scans. In that study,average discrepancy rates were obtained from individualinterpretations of residents and fellows compared withattending physicians. The lowest rate of 1.2% was ob-tained from fellow interpretations. This was attributedpartially to the higher level of training and experience offellows and partially to the known availability of pertinentpatient history at the time of interpretation. This is plausi-ble because there is an increased likelihood of an accurateinterpretation if pertinent medical history is available (9).In this series, we describe a discrepancy rate of 1% (134/12,600) between resident and attending interpretations ofchest radiographs for the presence or absence of pneumo-nia. This is at the low end of reported discrepancy rates.The recent literature has focused on cross-sectional imaging,

Figure 1. A 74 year-old woman presented to the emergency depgraphs initially were interpreted by the resident as normal. The staThe panel interpretation was in agreement with the staff radiologismiddle lobe with loss of the right heart border (silhouette sign) and

rather than plain films. The relatively low discrepancy rate

described in this series may be attributed to the difference inmodalities. However, to our knowledge, the literature doesnot support the premise of a lower error rate for interpreta-tion of plain films compared with cross-sectional modalities.

The majority of amended reports in the present studywere interpreted initially by residents in their second yearof training in radiology (PGY-3). We did not review in-terpretations of all 12,500 additional chest radiographsinterpreted on call during the study period to identify thePGYs of interpreting residents. However, PGY-3 residentsare responsible for approximately two thirds of the on-callshifts in our institution and were responsible for 80% ofamended chest radiographic interpretations for pneumo-nia. In light of the small percentage of amended reportsgenerated for pneumonia, the difference attributable toresident experience is negligible.

Evidence regarding the impact of training level on theerror rate of residents compared with board-certified radi-ologists is mixed. In one series, beyond the level of fun-damental training, the error rate reflected individual char-acteristics of the radiologist, rather than level of training(19). Some studies reported an inverse relationship be-

ent with chest pain. Posteroanterior and lateral chest radio-iologist amended the report to right middle-lobe pneumonia.est radiographic findings include increased opacity in the rightronchograms.

artmff radt. Ch

tween resident training level and error rate. Wysoki et al.

661


(10) reported a greater discrepancy rate among third-yearradiology residents compared with first- and second-yearresidents in the interpretation of head CT scans in thesetting of trauma. They speculated that the greater errorrate of more experienced residents was caused by over-confidence. Other series also documented a greater errorrate among senior residents compared with their juniorcounterparts (12). However, Ginsberg et al. (13) revieweda series of CT angiograms for the evaluation of pulmo-nary embolism and found a high level of agreement be-tween fellow and attending physician interpretations. Per-haps fellows, who are practicing within their chosen sub-specialty, are more similar to attending physicians than to

Figure 2. A 30 year-old man presented to the emergency departmident interpretation of the chest radiograph was right middle-lobe ppanel was in agreement with the staff radiologist.

Table 2Agreement Between Preliminary Resident, Staff Radiologist, an

Interpreter Pneumonia Diagnosis No

Radiology residents 29 (29%)Staff radiologists 71 (71%)Panel 60 (60%)

residents.

662

In this study, when the three members of the panelwere presented with the series of 100 chest radiographsamended for the presence or absence of pneumonia, aunanimous opinion was reached in only 72%. The paneldiagnosis was a majority opinion in 28%. This illustratesthe difficulty of these cases and the variability that existseven in a panel of experienced radiologists. In many ofthe cases that resulted in an amended report, the diagnosiswas not clear cut. This highlights the added value ofpanel review for difficult cases.

There was greater agreement between the panel andthe attending physician issuing the amended report thanwith the resident preliminary report, with � values of 0.76

with symptoms of upper respiratory tract infection. The initial res-onia. The staff radiologist amended the report to normal. The

nel Interpretation

monia Diagnosis � Agreement (95% confidence interval)

1 (71%) 0.428 (0.261–0.595)9 (29%) 0.760 (0.626–0.894)0 (40%)

entneum

d Pa

Pneu

724

and 0.428, respectively. � Values ranging from 0.3 to


greater than 0.85 have been reported in prior studies, withgreater agreement values seen between attending physi-cians and fellows than between attending physicians andresidents (11,13,18). There is evidence that panel and/ormultiple readings of radiographs, especially difficult ones,contribute significantly to increased sensitivity and reducederror rates for various modalities (14,15,23). Hillman et al.(16) reported increased accuracy of interpretation afterconsultation. In that study, a series of films was presentedto groups of three radiologists; when at least two of threeinitially agreed on a diagnosis, they were correct morefrequently than not. In academic institutions such as ours,presentation of amended cases to groups of residents atteaching conferences is standard practice and may be ben-eficial in the reduction of resident error rate.

In these series of 100 chest radiographs, the panel di-agnosed 60 cases of pneumonia. There is limited litera-ture describing the distribution and extent of missed pneu-monias in an emergency department practice. We foundthat the majority of chest radiographs diagnosed to bepneumonia by the panel were subsegmental or smaller,with a slight right-sided predominance. Although it isexpected that smaller subtler abnormalities more likelywould be missed on an initial resident interpretation, it issomewhat surprising that more right-sided pneumoniaswere missed. There is a popular belief that left-sidedchest radiographic abnormalities are more likely to bemissed because of the presence of the heart and retrocar-diac vascular structures, which may confuse the less ex-perienced interpreter. However, when left-sided patho-logic states were missed, the lower lobe was the mostpredominant site.

A major objective of this study is to characterize com-mon features of difficult cases that lead to the generationof amended reports. This information may be used in the

Table 3Characteristics of Pneumonia for the 60 Cases Diagnosed bythe Panel

Finding No. of Patients (%)

Increased opacity 60 (100)Loss of vascular markings 43 (72)Silhouette sign 24 (40)Air bronchogram 22 (37)Linear opacity 12 (20)Nodular opacity 11 (18)Lymphadenopathy 1 (2)

training of residents by systematizing common findings

that were routinely overlooked, thus serving as an aid toreduce “missed pneumonias” on chest radiographs. Astrong understanding of the range of normal and abnor-mal, as routine as it may appear, has been clearly shownto reduce the variability in interpreting chest radiographs(17). Among the 60 cases of pneumonia diagnosed by thepanel, 100% (60/60) were characterized by increasedopacity; 72% (43/60), by the loss of normal vascularmarkings; and 40% (24/60), by the silhouette sign. Airbronchograms, a finding not easily recognized by nonradiologists (20), was present in 37% (22/60). Forty-threepercent (21/49) were conspicuous on only one projection.Residents may be able to improve their accuracy in diag-nosing pneumonia by becoming familiar with these subtlefindings. In our institution, subtle chest radiographic find-ings of pneumonia form the basis of regular teachingconferences.

This study has limitations. Generation of amended re-ports depends on attending initiation. This study relied onthe assumption that a report was filed for every case inwhich there was a discrepancy between attending andresident interpretations. We did not review chest radio-graphs of the nearly 12,500 cases that were not amendedfor the diagnosis of pneumonia and have no data regard-ing those cases. Additionally, only 75% (100/134) of ra-diographs were available for review. In this study, wechose to rely on a panel of three experienced radiologiststo diagnose pneumonia based solely on chest radiographiccriteria. Although this serves as an imperfect gold stan-dard, it simulates the setting in which residents interpretchest radiographs, with a paucity of clinical data, in theemergency department setting.

In conclusion, resident interpretation of chest radio-graphs for pneumonia has a low error rate. Missed pneu-monias often were segmental or smaller and conspicuouson only one projection.

ACKNOWLEDGMENT

The authors thank E. Stephen Amis Jr, MD, our chair-man, for conceiving of this project and supporting itscompletion.

REFERENCES

1. National Vital Statistics Reports. Vol 52, No. 9. Atlanta, GA: Centers forDisease Control and Prevention, 2003.

2. Clinical Classifications for Health Policy Research: Hospital InpatientStatistics, 1996. Rockville, MD: Agency for Health Care Policy and Re-search, 1999. AHCPR publication no. 99-0034.

3. Marnee TJ. Community-acquired pneumonia. Clin Infect Dis 1994; 18:501–515.

663


4. Barlett JG, Mundy LM. Community-acquired pneumonia. N Engl J Med1995; 333:1618–1624.

5. Wipf JE, Lipsky BA, Hirshmann JV, et al. Diagnosing pneumonia byphysical examination: relevant or relic? Arch Intern Med 1999; 159:1082–1087.

6. Gatt ME, Spectre G, Paltiel O, et al. Chest radiographs in the emer-gency department: is the radiologist really necessary? Postgrad Med J2003; 79:214–217.

7. Carney E, Kempf J, DeCarvalho V, et al. Preliminary interpretations ofafter-hours CT and sonography by radiology residents versus final in-terpretations by body imaging radiologists at a level 1 trauma center.AJR Am J Roentgenol 2003; 181:367–373.

8. Wechsler RJ, Spettell CM, Kurtz AB, et al. Effects of training and expe-rience in interpretation of emergency body CT scans. Radiology 1996;199:717–720.

9. Aideyan UO, Berbaum K, Smith WL. Influence of prior radiologic infor-mation on the interpretation of radiographic examinations. Acta Radiol1995; 2:205–208.

10. Wysoki MG, Nassar CJ, Koenigsberg RA, et al. Head trauma: CT scaninterpretation by radiology residents versus staff radiologists. Radiology1998; 208:125–128.

11. Lowe LH, Draud KS, Hernanz-Schulman M, et al. Nonenhanced limitedCT in children suspected of having appendicitis: prospective compari-son of attending and resident interpretations. Radiology 2001; 221:755–759.

12. Lal RL, Murray UM, Eldevik OP, et al. Clinical consequences of misin-terpretations of neuroradiologic CT scans by on-call radiology resi-
dents. AJNR Am J Neuroradiol 2000; 21:124–129.
664

13. Ginsberg MS, King V, Panicek DM. Comparison of interpretations ofCT angiograms in the evaluation of suspected pulmonary embolism byon-call radiology fellows and subsequently by radiology faculty. AJRAm J Roentgenol 2004; 182:61–66.

14. Markus JB, Somers S, O Malley BP. Double contrast barium enema:effect of multiple readings on perception error. Radiology 1990; 175:155–156.

15. Hessel SJ, Herman PE, Swenson RG. Improving performance bymultiple interpretations of chest radiographs. Radiology 1978; 127:589 –594.

16. Hillman BJ, Swensson RG, Hessel SJ, et al. The value of consultationamong radiologists. AJR Am J Roentgenol 1976; 127:807–809.

17. Potchen EJ, Cooper TG, Sierra AE, et al. Measuring performance inchest radiography. Radiology 2000; 217:456–459.

18. Melbye H, Dale K. Interobserver variability in the radiographic diagnosisof adult outpatient pneumonia. Acta Radiol 1992; 33:79–81.

19. Herman PG, Hessel SJ. Accuracy and its relationship to experience inthe interpretation of chest radiographs. Invest Radiol 1975; 10:62–67.

20. Young M, Marrie TJ. Interobserver variability in the interpretation ofchest roentgenograms of patients with possible pneumonia. Arch InternMed 1994; 154:2729–2732.

21. Albaum MN, Hill LC, Murphy M, et al. Interobserver reliability of thechest radiograph in community-acquired pneumonia. Chest 1996; 110:343–350.

22. Nathanson N. Emergency Radiology Credentialing Examination: 4 yearexperience, Emerg Radiol 1995; 2:245–247.

23. Nodine CF, Kundel HL, Mello-Thoms C, et al. How experience and
training influence mammography expertise. Acta Radiol 1999; 6:575–585.

radiology residents’ on-call interpretation of chest radiographs for pneumonia1

Documents