Er

RMa

b

6c

d

a

ARRA

KCRRSR

1

aanfi

d((

0h

European Journal of Radiology 83 (2014) 329– 337

Contents lists available at ScienceDirect

European Journal of Radiology

jo ur nal ho me page: www.elsev ier .com/ locate /e j rad

valuating data capture methods for the establishment of diagnosticeference levels in CT scanning

achael E. Moorina,b,1, David A.J. Gibsonb,∗, Rene K. Forsythc,1,ax K. Bulsarad,2, C. D’Arcy J. Holmanb,3

Centre for Population Health Research, Faculty of Health Science, Curtin University, GPO Box U1987, Perth, Western Australia 6845, AustraliaCentre for Health Services Research, School of Population Health, University of Western Australia, 35 Stirling Highway, Crawley, Perth, Western Australia009, AustraliaDepartment of Medical Imaging Science, Curtin University, GPO Box U1987, Perth, Western Australia 6845, AustraliaInstitute of Health Research, University of Notre Dame, 19 Mouat Street, Fremantle, Western Australia 6959, Australia

r t i c l e i n f o

rticle history:eceived 23 August 2013eceived in revised form 30 October 2013ccepted 7 November 2013

eywords:T scanadiation dosageadiation dosimetryurvey methodsadiology information systems

a b s t r a c t

Objective: Concerns about the radiation dose associated with CT scanning have led to a call for estab-lishment of diagnostic reference levels. Self-complete surveys have been used extensively to gather thisinformation, however, departmental Radiological Information System’s/Picture Archive CommunicationSystems (RIS/PACS) also hold this information. We compared dosimetry derived from survey with thatusing RIS/PACSs.Methods: Technical data were collected from a large metropolitan tertiary hospital in WA using both datacollection methods for a range of adult CT scanning examinations. Radiation dose was calculated fromboth datasets and the results evaluated for several indexes of inter-rater agreement.Results: Radiation dose calculated using self-report survey data differed both systematically and propor-tionally from that calculated using RIS/PACS data. Differences were not consistent across CT examinationtype and thus not amenable to simple correction. The disparity was greater and more variable for organdose than effective dose due to reliance of survey data on “generic” anatomical start and stop limitscompared with actual data available on RIS/PACS.

Conclusions: The bias observed in our study indicates that care should be taken when interpreting theresults of studies measuring radiation dose using self-complete surveys. The availability of electronicdatabases that include information required for the evaluation and monitoring of CT radiation dose pro-vides the opportunity to capture better quality data in a cost-effective manner. We recommend thatnational and local databases are established that routinely capture these data so as to facilitate thedevelopment and monitoring of radiation dose associated with CT scanning.

. Introduction

CT scanning has always been recognised as a relatively high radi-tion dose procedure, but in the early days there was no viable

lternative. No other modality could compete with CT on the diag-ostic accuracy of brain scans; and when CT of the rest of the bodyrst began, its use was largely limited to cancer patients where

∗ Corresponding author. Tel.: +61 8 6488 1261.E-mail addresses: r.moorin@curtin.edu.au (R.E. Moorin),

avid.gibson@uwa.edu.au (D.A.J. Gibson), rene.forsyth@postgrad.curtin.edu.auR.K. Forsyth), max.bulsara@nd.edu.au (M.K. Bulsara), darcy.holman@uwa.edu.auC.D.J. Holman).

1 Tel.: +61 8 9266 1854.2 Tel.: +61 8 9433 0297.3 Tel.: +61 8 6488 1261.

720-048X/$ – see front matter © 2013 Elsevier Ireland Ltd. All rights reserved.ttp://dx.doi.org/10.1016/j.ejrad.2013.11.003

© 2013 Elsevier Ireland Ltd. All rights reserved.

radiation dose was less of a concern given the balance of risks [1].Today, CT is used extensively in benign disease and young patients,for whom radiation protection considerations hold greater weight[1,2].

Awareness and concern regarding the high burden of radiationdose to the population stemming from CT began in the late 1980sfollowing a survey conducted by the UK National RadiologicalProtection Board [3]. It found, despite comprising only 2% of allmedical imaging examinations, CT contributed a disproportion-ately large proportion (20%) of the collective population dose fromdiagnostic imaging [3,4]. The survey also found radiation dosesfrom CT was trending in a direction at odds with the decreas-

ing collective patient doses received from conventional x-rayexaminations [3]. Rather, CT examinations had grown steadily tobecome a significant source of population exposure from medicalradiology [4]. A follow-up study showed CT in the UK had doubled

3 rnal o

btSa[d6tOnohif

co(tcbCtsesn

lmiipap[upet

iidsi

2

lutsa

123456

30 R.E. Moorin et al. / European Jou

oth its contribution to the population effective dose (40%) andhe proportion of all radiological examinations (4%) by 1995 [5].imilar findings were reported in other developed countries suchs Switzerland, [6] Germany, [7] the Netherlands [8] and the US9]. Mettler et al. [9] found CT in the USA accounted for 11% ofiagnostic radiological procedures in 1999, compared with only% in 1990. This study also showed CT accounted for 67% of theotal effective radiation dose from diagnostic imaging in 1999 [9].f particular concern, Mettler et al. [9] found 11% of CT exami-ations were performed on children. In 1998, the Royal Collegef Radiologists (UK) estimated CT contributed approximately onealf of the collective dose from all examinations, with the average

n UK CT workload to be a 24% increase per year against only 2%or all other medical and dental radiological procedures [10].

Concerns have been expressed about the radiation dose asso-iated with CT scanning, which have led to guidelines, advisingn clinical indications for utilisation or diagnostic reference levelsDRLs) for the radiation dose received from each type of examina-ion [1,5,11]. Due to paucity of data, estimates of radiation dose andancer risk from CT scanning to underpin the creation of DRLs haveeen undertaken using self-report surveys combined with ‘typical’T protocol/machine settings [12–14]. However, this data cap-ure technique has inherent limitations typical of all self-completeurveys such as response bias, low response fraction, poor gen-ralizability (due to the data captured not being a representativeample either purposefully or otherwise) and, if the instrument hasot been appropriately validated, poor validity and reliability.

Databases of CT scanner radiation dose facilitated by Radio-ogical Information Systems (RIS) and Picture Archiving Com-

unication Systems (PACS) could enable comprehensive routinenstitutional benchmarking, optimisation of CT protocols and qual-ty control [15]. These databases would also enable more accurateatient specific dose estimation than possible from data sourcesvailable previously, which would ultimately lead to more accurateatient specific risk assessment to better inform imaging decisions15,16]. However, while the data required are captured in individ-al department RIS/PACS, extracting these data efficiently remainsroblematic. Thus self-report survey methods are currently usedxtensively for capture of national and regional CT dose informa-ion.

The aim of this study was to compare the concordance (reliabil-ty and validity) of the radiation dose derived from a recognisednternational self-complete survey data capture instrument withose obtained from data extracted from RIS/PACS (i.e. the goldtandard) for a range of CT scanning examination types undertakenn the same public hospital.

. Materials and methods

Technical data on local CT practice were collected from aarge metropolitan tertiary (teaching) hospital in Western Australiasing (i) a self-complete survey and (ii) information contained inhe dose report obtained from the PACS database for diagnostic CTcanning examinations performed on adults using the same makend model 64 slice scanner in the following six clinical scenarios.

. Routine head for trauma or stroke.

. Chest for lung cancer (known or suspected metastases).

. Chest for pulmonary embolism.

. High resolution chest for chronic obstructive pulmonary disease.

. Abdomen/pelvis (abscess).

. Chest/abdomen/pelvis for lymphoma staging or follow up.

f Radiology 83 (2014) 329– 337

This study was approved by the Western Australia Departmentof Health Human Research Ethics Committee, which exempted thestudy from requiring individual patient consent.

2.1. Data collection: survey data

Technical data were collected by means of a questionnaire com-pleted by the chief CT radiographer in early 2011. The questionnairewas identical to the survey used by the national survey of dosesfrom CT in the UK in 2003 [12], amended only in terms of the clini-cal scenarios examined (addition of ‘chest’ for pulmonary embolusand removal of ‘abdomen’ for liver metastases) and limited toexaminations performed on adults. The questionnaire sought tech-nical data in relation to examinations performed on ‘average-sized’patients to represent ‘usual practice’. Technical information wascollected for each clinical scenario (excluding the scout view) con-sisting of separate scanning sequences (phases) where appropriate,each representing a single helical exposure or a series of similaraxial exposures using identical scan conditions. The data includedvarious technical parameters such as kilovoltage (kV), milliamper-age (mA), tube rotation times, collimation widths, pitch, scanningmethod and typical anatomical reference start–stop positions ofthe scan. Respondents were asked to report the average volumeweighted CT dose index (CTDIvol) and dose–length product (DLP)based on up to ten standard (‘average-sized’) adult patients under-going each examination type, as performed by the 2003 UK dosesurvey [12].

2.2. Data collection: Picture Archiving Communication System

For each scanning scenario, data pertaining to a random sampleof 20 adult cases (or all the cases available if less than 20) identi-fied from the departmental PACS data base, performed between1st January and 30th April 2011 inclusive, were collected bythe researchers. Protocol information (excluding the scout view)consisted of separate scanning sequences (phases) where appropri-ate for each protocol. The data collected included various technicalparameters such as kilovoltage (kV), milliamperage (mA), tuberotation times, collimation widths, pitch, scanning method, volumeweighted CT dose index (CTDIvol) and dose–length product (DLP)for each case. CT radiographers were interviewed to acquire typicalanatomical start–stop locations for each scanning scenario.

2.3. Radiation dosimetry

For both sets of data (survey and PACS), dosimetry involvedreporting of CTDIvol, DLP values and calculation of organ andeffective dose for each sequence using scan settings providedby the data. Scanner-specific dosimetrics published by ImPACTare included within the CT patient dosimetry calculator [17]. Forsequences performed with automatic tube current modulation,doses were calculated using reported values, where available, of(average) tube current or current–time product (mAs) includingthe effects of modulation. For survey data the reported anatomicalstart–stop locations were used to determine the placement ofthe scanning field on the ImPACT dosimetry calculator. However,for the PACS data the average scan length of the cases collected,obtained by dividing DLP by CTDIvol, was used to modify thereported anatomical start stop locations for each clinical scenario.For both data collections, in circumstances when ImPACT cal-culated values of CTDIvol differed to those reported, the organand effective doses were corrected by the ratio of reported and

the ImPACT calculated CTDIvol. For example, if the reportedCTDIvol was 20 mGy but the ImPACT software calculated 18 mGy(from the imputation of other scanning parameters) then thesubsequent effective dose values were multiplied by 1.11 (20/18).

rnal of Radiology 83 (2014) 329– 337 331

Pcr2ae

2

ddfwbb

taffitrweatpttf(oi<a

dPwpuott

tlrtr

3

iCesoovdTp

of

CT

scan

nin

g

par

amet

ers

and

rad

iati

on

dos

imet

ry

rep

orte

d

usi

ng

self

-com

ple

te

surv

ey

wit

h

dat

a

extr

acte

d

from

dep

artm

enta

l pic

ture

arch

ival

com

mu

nic

atio

n

syst

em.

n

Self

-rep

ort

surv

ey

dat

a

Dat

a

cap

ture

d

from

the

dep

artm

enta

l pic

ture

arch

ive

com

mu

nic

atio

n

syst

em

An

atom

ical

pos

itio

ns

Seqa

kVb

Pitc

hc

CTD

Ivol

dD

LPe

Nu

mbe

r

of

case

s

Seqf

kVg

Pitc

hh

CTD

Ivol

im

ean

(StD

ev)

DLP

jm

ean

(StD

ev)

Top

Bot

tom

roke

or

trau

ma

Top

of

hea

d

Bas

e

of

sku

ll

1

120

0.42

3

86.7

0

1256

20

1

120

0.42

3

64.6

(28.

6)

1207

.8

(647

.3)

ng

can

cer

Abo

ve

lun

g

apic

esB

ase

of

lun

g/ad

ren

als

1

120

1.07

8

26.6

0

1131

20

2

120

1.07

8

22.5

(11.

1)

404.

9

(458

.6)

ulm

onar

y

embo

li

Abo

ve

lun

g

apic

es

Bas

e

of

lun

g/ad

ren

als

1

120

0.67

3

35.2

0

968

20

3

120

0.67

3

47.8

(23.

2)

738.

2

(329

.8)

ron

ic

obst

ruct

ive

pu

lmon

ary

dis

ease

Abo

ve

lun

g

apic

es

Bas

e

of

lun

g/ad

ren

als

1

140

0.51

6

38.1

0

1048

20

2

140

0.51

6

13.7

(9.4

)

489.

0

(340

.9)

elvi

s

for

live

r

met

asta

ses

Dia

ph

ragm

Sym

ph

ysis

pu

bis

1 12

0

0.89

126

.70

1226

20

1

120

0.89

1

19.6

(8.7

)

903.

4

(382

.7)

men

and

pel

vis

for

lym

ph

oma

Abo

ve

lun

g

apic

es

Sym

ph

ysis

pu

bis

1 12

0

1.07

8

24.8

0

1720

10k

2

120

1.07

8

27.6

(6.1

)

431.

6

(832

.4)

rep

orte

d

nu

mbe

r

of

sequ

ence

s

(ph

ases

)

usu

ally

un

der

take

n

for

the

exam

inat

ion

.ep

orte

d

kilo

volt

age

usu

ally

use

d

or

the

exam

inat

ion

.rt

ed

pit

ch

(mm

)

usu

ally

use

d

for

the

exam

inat

ion

.se

lf-r

epor

ted

aver

age

volu

met

ric

CT

Dos

e

Ind

ex, m

easu

red

in

mG

y

for

the

exam

inat

ion

.re

por

ted

aver

age

dos

e

len

gth

pro

du

ct, m

easu

red

in

mG

y

cm

for

the

exam

inat

ion

.ia

n

nu

mbe

r

of

sequ

ence

s

(ph

ases

)

obse

rved

for

the

case

s

coll

ecte

d.

ilov

olta

ge

obse

rved

acro

ss

the

case

s

coll

ecte

d.

itch

(mm

)

obse

rved

acro

ss

the

case

s

coll

ecte

d.

and

ard

dev

iati

on)

of

the

volu

met

ric

CT

dos

e

ind

ex

(mG

y)

obse

rved

for

the

case

s

coll

ecte

d. N

ote

wh

ere

mu

ltip

le

sequ

ence

s

(ph

ases

)

wer

e

obse

rved

the

CTD

Ivol

for

the

exam

inat

ion

was

the

sum

of

the

sequ

ence

vol.

and

ard

dev

iati

on) o

f th

e

dos

e

len

gth

pro

du

ct

(mG

y cm

) obs

erve

d

for

the

case

s

coll

ecte

d. N

ote

wh

ere

mu

ltip

le

sequ

ence

s

(ph

ases

) wer

e

obse

rved

the

DLP

for

the

exam

inat

ion

was

the

sum

of

the

sequ

ence

spec

ific

DLP

.as

es

wer

e

iden

tifi

ed

over

the

dat

a

cap

ture

tim

e fr

ame.

lect

ion

s

wer

e

base

d

on

exam

inat

ion

s

per

form

ed

on

the

sam

e

mod

el

and

mak

e

of

CT

scan

ner

.

R.E. Moorin et al. / European Jou

rotocol (cumulative) values of DLP and effective dose were alsoalculated on the basis of summation over all routine sequenceseported for each clinical scenario (Sequence 1 DLP + Sequence

DLP + Sequence 3 DLP = Protocol DLP). For the PACS data theverage and standard deviation of the CTDIvol, DLP organ andffective doses of the cases collected were reported.

.4. Analysis of inter-rater agreement

Evaluations of the agreement of organ dose and effective doseerived using PACS versus self-reported survey data were con-ucted under three scenarios: (i) the radiation doses calculateddrom PACS and survey data both adjusted to one decimal place; (ii)here the two values differed by ≤1 the PACS value was reset to

e equal to survey values; and (iii) where the two values differedy ≤5 the PACS value was reset to be equal to survey values.

The reliability (inter-rater agreement) of the two data collec-ion instruments as input to radiation dosimetry calculations wasssessed using the intraclass correlation coefficient (ICC) separatelyor each clinical scenario. We used the ICC in two distinct ways. Therst ICC evaluated the absolute agreement of the organ and effec-ive doses produced for single measures, this is the reliability of theatings for one, typical, single dose [18,19]. The second ICC used wasith respect to average measures, the reliability of all the organ and

ffective dose produced by the different data collection instrumentsveraged together [18,19]. In addition, the concordance correla-ion coefficient (�c) was used to determine the degree to whichairs of organ/effective doses resulting from the two data collec-ion instruments related to a line of best fit (set to 45◦ throughhe origin) [20]. The data were evaluated for both precision (howar each observation deviated from the best-fit line) and accuracyhow far the best-fit line deviated from the 45◦ line through therigin). The values of the concordance correlation coefficient werenterpreted using a descriptive scale suggested by McBride [21]:0.9: poor; 0.90–0.95: moderate; 0.95–0.99: substantial and >0.99lmost perfect.

The equality of measurements (i.e. radiation doses) from twoifferent analytical methods (PACS versus survey) was tested usingassing and Bablok regression. This is a linear regression procedureith no special assumptions regarding the distribution of the sam-les and the measurement errors [22]. This regression model wassed to evaluate if the radiation doses calculated by the two meth-ds differed by a constant amount (systematic differences) and forhe presence of proportional differences in the values produced byhe two methods.

Finally, Bland–Altman plots were used to graphically comparehe radiation doses produced using data from the two data col-ection methods [23]. For this study the differences between theadiation doses produced by the two methods were plotted againsthe radiation doses produced by PACS data since this method is theeference or “gold standard” [24].

. Results

As shown in Table 1 the CT technical information for the six clin-cal scenarios investigated varied in terms of number of sequences,TDIvol and DLP across the data collection methods. Data on actualxaminations undertaken, sourced from the PACS, showed that allcenarios except ‘head’ and ‘abdomen/pelvis’ had a median numberf sequences greater than one. Only a single sequence was reportedn the self-complete survey for all scenarios. Comparison of CTDI-

ol showed that the average across the cases collected using PACSata was substantially lower than self-reported in three scenarios.he greatest difference occurred in ‘chest’ for chronic obstructiveulmonary disease with a CTDIvol reported nearly three times Ta

ble

1C

omp

aris

on

Exam

inat

io

Hea

d

for

stC

hes

t

for

luC

hes

t

for

pC

hes

t

for

chA

bdom

en/p

Ch

est,

abd

o

aSe

q,

self

-b

kV, s

elf-

rc

Self

-rep

od

CTD

Ivol

,

eD

LP, s

elf-

fSe

q,

med

gM

edia

n

kh

Med

ian

pi

Mea

n

(st

spec

ific

CTD

Ij

Mea

n

(st

kO

nly

10

cB

oth

dat

a

col

3 rnal o

lu

ppsmpn

fdstrmh

di‘swtodicw4

tstfompiof‘c

eempdtt(vitttmefiPttdct

32 R.E. Moorin et al. / European Jou

arger than the PACS mean CTDIvol. The DLP was consistently lowersing the average of the PACS data compared with self-report.

Table 2 presents the organ and effective radiation dose com-uted using the two methods of data collection. The PACS dataroduced substantially lower (>2 mSv) effective doses in threecenarios (‘chest’ for pulmonary embolism, chronic obstructive pul-onary disorder and ‘abdomen/pelvis’). Conversely, the PACS data

roduced substantially higher (>2 mSv) effective doses in two sce-arios (‘chest’ for lung cancer and ‘chest/abdo/pelvis’).

The individual organ doses did not follow the pattern describedor the whole body effective dose. PACS data produced higheroses in organs near the limits of the scanning field, because thecan lengths observed in the PACS data were substantially longerhan those calculated using the self-reported anatomical start-stopeference points. For example, in the ‘chest’ for lung cancer and pul-onary embolism scenarios the radiation dose to the thyroid was

igher using the PACS data by 38.9 and 29.7 mGy, respectively.Table 3 presents the analysis of agreement between the two

ata capture methods. There was moderate agreement, accord-ng to the ICC and concordance (pc) for some scenarios, such ashead’, ‘abdo/pelvis’ and ‘chest/abdo/pelvis.’ In all three of thesecenarios the precision (P) was high (≥0.98), but the accuracy (Cb)as only moderate. Both the ICC and concordance were poor in

he remaining three scenarios. Particularly poor agreement wasbserved on the ‘chest’ for chronic obstructive pulmonary disor-er scenario, where the ICC and pc were <0.6. While the precision

n this scenario was perfect (1.00), the accuracy was only 0.56, indi-ating that the pairs of organ doses produced by the two methodsere strongly linearly related, but departed substantially from the

5◦ line of concordance.Differences in the typology of the agreement was reflected in

he regression analysis (Table 4), which evaluated the degree ofystematic and proportional difference observed in the dosime-ry produced using the two data capture methods. In the ‘chest’or chronic obstructive pulmonary disorder scenario, there werenly non-significant minor systematic differences between the twoethods (A = 0.00 [−0.10 to 0.056]). There were significant pro-

ortional differences (B = 2.64 and 95% confidence interval did notnclude 1), indicating a non-constant significant difference in thergan dose produced. In contrast, there were no systematic dif-erences and only non-significant proportional differences in thehead’ scenario, indicating the dosimetry produced by the two dataapture methods were in good agreement.

When all the scenarios were evaluated together (either usingffective dose or organ dose) the agreement was poorer than whenvaluated individually. This is shown in both Tables 3 and 4, but isost apparent in Fig. 1. Panel A of Fig. 1 shows the Bland–Altman

lot for effective dose comparing the differences between the twoata capture methods with the dosimetry produced using PACS (i.e.,he reference or gold-standard method). In Figs. 1 and 2, horizon-al lines are drawn at the mean difference the limits of agreement±1.96 standard deviation) and represent the 95% confidence inter-als around the agreement. The Bland–Altman plot can assist indentifying relationships between the differences and the magni-ude of measurements (between the survey and PACS data). Theechnique aids in identifying the types of systematic bias to aid inhe identification of potential corrections. It can be seen that the

ean (−1.7) of the comparison did not lie at zero indicating nonqual variances; however, all the points lay within the 95% con-dence intervals so there was no evidence of a systematic error.anel B of Fig. 1 shows the Bland–Altman plot for organ doses withhose from each scenario identified by a different symbol. While

he mean lay closer to the zero line (−1.1) than in panel A, theata points followed different gradients across scenarios. This indi-ated that the relationship between the dosimetry produced bywo data capture methods had systematic bias. For example, the

f Radiology 83 (2014) 329– 337

‘chest’ (chronic obstructive pulmonary disease) values in panel Cof Fig. 2 had a steep negative gradient indicating bias. The ‘chest’(pulmonary embolus) values in panel B of Fig. 2 fanned out, indicat-ing systematic bias where the variation in measures were for largerPACS generated values.

4. Discussion

The results of our study show radiation dosimetry calculationsdiffered, both systematically and proportionally, when using self-report survey data and PACS data sources. Importantly, the studyfound the differences were not consistent across CT examinationtypes, indicating the disparity cannot be adjusted using a sim-ple correction factor. The disparity was greater and more variablefor organ dose than effective dose, most likely resulting from thereliance of the survey data on ‘standardised’ anatomical start andstop limits compared with actual start and stop limits in the PACSdata. The PACS data also provided information regarding the vari-ance of the radiation dose unavailable using the survey instrument,because the survey instrument required the respondent to providea single CTDIvol and DLP measure averaged from 10 ‘average’ adultpatients, whereas the PACS data provided individual level data foreach randomly selected case.

The observed discordance between the organ dose calculatedfrom each data source is an important finding since individual orpopulation risk estimates are derived using organ doses rather thaneffective dose [25]. Controversies persist about the effect of low-level radiation exposure and how these risks should be reported,especially to the public; however, it is generally accepted that thereis no safe level of exposure (i.e., the linear-no-threshold model)[25]. In light of concerns regarding the increasing use of CT scan-ning and the expansion of its use in high risk populations such aschildren, several studies have attempted to quantify the risk to thepopulation of CT scanning based upon radiation doses generatedfrom surveys [13,26]. In light of our results demonstrating the dis-cordance of the data collected using self-complete surveys with amore rigorous data source, care should be taken when interpretingthe results of these studies.

In contrast, guidance regarding the establishment of DRLs rec-ommend they be based on the 75th percentile of the radiation dosemeasured across providers’ derived dose data from a minimumof 10 standard sized patients undergoing standard examinations[27]. Historically DRLs have been articulated in terms of both theeffective dose and the DLP. DLPs have to date been generated exclu-sively from self-complete surveys [12,28,29] and, while our studyhas shown the data are not concordant with data obtained fromPACS, the bias with respect to effective dose does appear to followa more consistent pattern allowing the potential for correction tobe applied.

4.1. Strengths and limitations

This study used a survey instrument applied extensively in theUnited Kingdom for collection of the self-report data [12], withoutmodification (except in terms of the clinical; scenarios examined)and is also recommended by the European Commission on Radia-tion Protection [30]. It represents a widely accepted example of thisdata collection method. Data from the PACS were extracted man-ually from the same institution and over the same time period asthe survey. In addition, all CT examinations included in the studywere undertaken on the same make and model of CT scanner and by

the same technical staff. Data extracted from PACS were a randomsample of 20 examinations on adult patients (i.e. double the num-ber recommended for generation of DRLS [27]) and were limitedto examinations most closely matched the clinical scenarios on the

R.E.

Moorin

et al.

/ European

Journal of

Radiology

83 (2014) 329– 337333

Table 2Organ radiation dose (mGy) and whole body effective radiation dose (mSv) of CT scanning protocols obtained using self-complete survey and data extracted from departmental picture archival communication system as inputto radiation dosimetry calculations.

Head (stroke) Chest (lung cancer) Chest (pulmonary embolism) High resolution chest (COPD) Abdomen/pelvis (metastases) Chest/abdo/pelvis(lymphoma staging)

Organ dose (mGy)a Diffb Organ dose (mGy)a Diffb Organ dose (mGy)a Diffb Organ dose (mGy)a Diffb Organ dose (mGy)a Diffb Organ dose (mGy)a Diffb

Self-reportc PACSd Self-reportc PACSd Self-reportc PACSd Self-reportc PACSd Self-reportc PACSd Self-reportc PACSd

Gonads 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.1 0.0 0.0 0.0 0.0 19.0 17.0 −2.0 16.0 20.0 4.0Bone marrow 4.3 3.8 −0.5 9.6 11.4 1.8 13.0 10.1 −2.9 15.0 5.8 −9.2 16.0 13.5 −2.5 23.0 26.4 3.4Colon 0.0 0.0 0.0 0.2 0.8 0.7 0.2 0.4 0.2 0.3 0.1 −0.2 34.0 26.0 −8.0 29.0 35.0 6.0Lung 0.2 0.2 0.0 36.0 34.0 −2.0 48.0 31.3 −16.7 55.0 20.8 −34.2 9.6 14.3 4.7 39.0 43.6 4.6Stomach 0.0 0.0 0.0 3.0 14.5 11.5 4.0 9.0 5.0 6.0 2.7 −3.3 38.0 28.6 −9.4 37.0 41.0 4.0Bladder 0.0 0.0 0.0 0.0 0.3 0.3 0.0 0.0 0.0 0.0 0.0 0.0 37.0 29.0 −8.0 29.0 38.6 9.6Breast 0.0 0.0 0.0 32.0 27.4 −4.6 43.0 27.0 −16.0 47.0 17.7 −29.3 2.0 8.9 6.9 32.0 35.0 3.0Liver 0.0 0.0 0.0 4.6 18.7 14.1 6.1 13.4 7.3 9.7 4.3 −5.4 36.0 27.2 −8.8 36.0 39.9 3.9Oesophagus 0.1 0.1 0.0 44.0 37.9 −6.1 59.0 35.2 −23.8 64.0 23.2 −40.8 1.7 3.1 1.4 43.0 47.6 4.6Thyroid 2.9 4.4 1.5 5.4 44.3 38.9 7.2 36.9 29.7 12.0 4.7 −7.3 0.1 0.2 0.1 8.2 18.0 9.8Skin 4.6 3.9 −0.7 6.5 9.6 3.1 8.7 7.7 −1.0 11.0 4.3 −6.7 11.0 10.2 −0.8 16.0 19.4 3.4Bone surface 18.0 15.4 −2.6 18.0 22.5 4.5 24.0 19.0 −5.0 29.0 11.1 −17.9 20.0 18.5 −1.5 35.0 40.2 5.2Brain 66.0 49.3 −16.7 0.3 1.3 1.0 0.4 0.6 0.3 0.6 0.2 −0.4 0.0 0.0 0.0 0.4 0.5 0.1Salivary glands (brain) 66.0 49.3 −16.7 0.3 1.0 0.7 0.4 0.6 0.3 0.6 0.2 −0.4 0.0 0.0 0.0 0.4 0.5 0.1Remainder organs 5.5 4.4 −1.1 8.8 16.1 7.3 12.0 12.9 0.9 14.0 5.4 −8.6 23.0 18.7 −4.3 28.0 32.8 4.8

Other organs of interestEye lenses 76.0 54.0 −22.0 0.3 0.7 0.4 0.0 0.5 0.5 0.6 0.3 −0.2 0.0 0.0 0.0 0.4 0.4 0.1Testes 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 8.7 5.4 2.0 4.6 2.6Ovaries 0.0 0.0 0.0 0.0 0.2 0.2 0.1 0.1 0.0 0.1 0.0 0.0 34.0 25.2 −8.8 31.0 35.2 4.2Uterus 0.0 0.0 0.0 0.0 0.2 0.1 0.0 0.1 0.1 0.1 0.0 0.0 36.0 27.1 −8.9 32.0 37.1 5.1Prostate 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 37.0 29.0 −8.0 29.0 38.6 9.6Whole body effective dose

(mSv)e2.9 2.4 −0.5 13.3 16.9 3.6 17.7 14.6 −3.1 20.4 7.8 −12.6 19.5 17.2 −2.3 28.9 33.6 4.7

a Organ dose reported in mGy – the amount of radiation received by the organ specified not modified by the tissue weighting factor for the organ.b Difference in radiation dose calculated using picture archival communication system (PACS) data compared with that calculated using self-report data. A negative number indicates radiation dose is lower using PACS data.c Organ and effective radiation dose calculated using the self-reported (survey) dose information.d Mean of the organ and effective radiation dose effective doses calculated across the cases using case specific scanning and dose information.e Effective dose reported in mSv–The whole body effective dose is the sum of the modified (using ICRP 103 tissue weighting factor for each organ) organ dose. This is a measure of the cancer risk to the individual due to ionising

radiation taking into account both the type of radiation and the radio-sensitivity of each organ being irradiated.Note that anatomical start and stop positions used during calculation of the radiation dose for survey data were those reported on the survey. For the PACS data CT staff were interviewed and the reported anatomical start stoplocations modified to match the average scan lengths for cases collected for each protocol.

334R

.E. M

oorin et

al. /

European Journal

of R

adiology 83 (2014) 329– 337

Table 3Evaluation of the agreement of organ dose and effective dose derived using picture archival communication systems (PACS) versus self-reported survey data.

Intra class correlationcoefficient (ICC)

All examinations Head CT Chest (Lung cancer) Chest (PE) Chest (COPD) Abdo/Pelvis Chest/Abdomen/Pelvis

Effective dose (mSv) Organ doses (mGy) Organ doses (mGy) Organ doses (mGy) Organ doses (mGy) Organ doses (mGy) Organ doses (mGy) Organ doses (mGy)ICCa 95% CI ICCa 95% CI ICCa 95% CI ICCa 95% CI ICCa 95% CI ICCa 95% CI ICCa 95% CI ICCa 95% CI

PACS vs surveySingle measuresb 0.81 0.18–0.97 0.84 0.78–0.89 0.95 0.86–0.98 0.75 0.47–0.89 0.78 0.53–0.91 0.58 0.14–0.81 0.91 0.76–0.97 0.94 0.19–0.99Average measuresc 0.90 0.30–0.99 0.92 0.88–0.94 0.97 0.92–0.99 0.86 0.64–0.94 0.88 0.69–0.95 0.73 0.25–0.90 0.95 0.86–0.98 0.97 0.32–0.99

PACS mod 1 vs surveySingle measuresb 0.81 0.18–0.97 0.84 0.78–0.89 0.95 0.86–0.98 0.75 0.47–0.89 0.78 0.53–0.91 0.57 0.15–0.81 0.91 0.76–0.97 0.94 0.12–0.99Average measuresc 0.90 0.30–0.99 0.92 0.88–0.94 0.97 0.92–0.99 0.86 0.64–0.94 0.88 0.70–0.95 0.73 0.26–0.90 0.95 0.86–0.98 0.97 0.21–0.99

PACS mod 5 vs surveySingle measuresb 0.83 0.30–0.97 0.85 0.79–0.89 0.94 0.85–0.98 0.76 0.50–0.90 0.79 0.53–0.91 0.57 0.15–0.81 0.93 0.83–0.97 0.96 0.86–0.98Average measuresc 0.91 0.46–0.99 0.92 0.88–0.94 0.97 0.92–0.99 0.87 0.67–0.95 0.88 0.70–0.95 0.73 0.26–0.90 0.96 0.90–0.99 0.98 0.92–0.99

Concordance correlationcoefficient (�c)

�c 95% CI �c 95% CI �c 95% CI �c 95% CI �c 95% CI �c 95% CI �c 95% CI �c 95% CI

PACS vs survey (pc) 0.78 0.14–0.96 0.84 0.79–0.89 0.94 0.92–0.96 0.74 0.47–0.88 0.78 0.57–0.89 0.56 0.43–0.66 0.91 0.84–0.94 0.94 0.88–0.97Precision (�) 0.81 0.86 1.00 0.77 0.82 1.00 0.98 0.99Accuracy (Cb) 0.96 0.98 0.94 0.96 0.94 0.56 0.93 0.95

PACS mod 1 vs survey (pc) 0.78 0.14–0.96 0.84 0.79–0.89 0.94 0.92–0.96 0.74 0.48–0.88 0.78 0.57–0.89 0.55 0.43–0.66 0.91 0.84–0.94 0.94 0.87–0.97Precision (�) 0.81 0.86 1.00 0.77 0.82 1.00 0.98 0.99Accuracy (Cb) 0.96 0.98 0.94 0.96 0.94 0.55 0.93 0.95

PACS mod 5 vs survey (pc) 0.81 0.17–0.97 0.85 0.79–0.89 0.94 0.91–0.96 0.76 0.50–0.89 0.78 0.57–0.89 0.55 0.43–0.66 0.93 0.86–0.96 0.95 0.89–0.98Precision (�) 0.84 0.87 1.00 0.78 0.82 0.99 0.97 0.97Accuracy (Cb) 0.97 0.98 0.94 0.97 0.95 0.56 0.96 0.99

PACS, Radiation dose values calculated using PACS data (to one decimal place).PACS mod 1, PACS dose values ≤1 reset to be equal to survey values.PACS mod 5, PACS dose values ≤5 reset to be equal to survey values.

a The intra class correlation coefficient measures the degree of absolute agreement among the radiation dose estimated using the two measurement instruments providing input data to the radiation dose calculation (PACS vssurvey).

b Estimates the reliability of single measurements.c Estimates the reliability of the averages of all measurements.

�c is the concordance correlation coefficient. It evaluates the degree to which pairs of observations (i.e. radiation dose calculated using PACS and survey data) fall on the 45◦ line (i.e. line of absolute agreement) through the origin.� is the Pearson correlation coefficient, which measures how far each observation deviates from the best-fit line, and is a measure of precision.Cb is a bias correction factor that measures how far the best-fit line deviates from the 45◦ line through the origin, and is a measure of accuracy.

R.E. Moorin et al. / European Journal of Radiology 83 (2014) 329– 337 335

Table 4Passing and Bablok regression testing the equality of the organ and effective doses calculated based on picture archival communication system (PACS) versus self-reportedsurvey CT examination data.

All examinations

Effective dose (mSv) Organ doses (mGy)

Regression equation y = 5.98 + 0.72x y = 0.00 + 1.06x

Point estimate 95% CI Point estimate 95% CI

Systematic differencesA* 5.98 −310.00 to 23.50 0.00 −0.031 to 0.00Proportional differencesB† 0.72 −0.40 to 20.67 1.06 0.91 to 1.17Random differencesRSD‡ 4.60 −9.02 to 9.02 6.45 −12.65 to 12.65Cusum test for linearity P = 0.99 P = 0.02§

Head CT Chest (lung cancer)

Organ doses (mGy) Organ doses (mGy)

Regression equation y = 0.00 + 1.22x y = −0.12 + 0.60x

Point estimate 95% CI Point estimate 95% CI

Systematic differencesA* 0 0.00 to 0.00 −0.12 −0.42 to −0.01Proportional differencesB† 1.22 1.13 to 1.34 0.60 0.27 to 0.89Random differencesRSD‡ 2.00 −3.92 to 3.92 7.80 −15.29 to 15.29Cusum test for linearity P = 0.05 P = 0.52

Chest (PE) Chest (COPD)

Organ doses (mGy) Organ doses (mGy)

Regression equation y = −0.12 + 1.16 x y = 0.00 + 2.64 x

Point estimate 95% CI Point estimate 95% CI

Systematic differencesA* −0.12 −0.47 to 0.00 0.00 −0.10 to 0.056Proportional differencesB† 1.16 0.68 to 1.54 2.64 2.57 to 2.69Random differencesRSD‡ 6.89 −13.51 to 13.51 0.28 −0.54 to 0.54Cusum test for linearity P = 0.25 P = 0.07

Abdomen/pelvis Chest/abdomen/pelvis

Organ doses (mGy) Organ doses (mGy)

Regression equation y = −2.13 + 1.35x y = −1.35 + 0.91x

Point estimate 95% CI Point estimate 95% CI

Systematic differencesA* −2.13 −5.04 to −0.07 −1.35 −3.50 to −0.04Proportional differencesB† 1.35 1.24 to 1.51 0.91 0.87 to 0.98Random differencesRSD‡ 1.99 −3.90 to 3.90 1.82 −3.56 to 3.56Cusum test for linearity P = 0.10 P = 0.36

Regression equation: the regression equation with the calculated values for A and B according to Passing and Bablok [23].A*Systematic differences. The intercept A is a measure of the systematic differences between the two methods. The 95% confidence interval for the intercept A tests thehypothesis that A = 0. This hypothesis is accepted if the confidence interval for A contains the value 0. If the hypothesis is rejected, then it is concluded that A is significantlydifferent from 0 and the methods differ by a constant amount.B†Proportional differences. The slope B is a measure of the proportional differences between the two methods. The 95% confidence interval for the slope B tests the hypothesisthat B = 1. This hypothesis is accepted if the confidence interval for B contains the value 1. If the hypothesis is rejected, then it is concluded that B is significantly differentfrom 1 and there is a proportional difference between the two methods.RSD‡Random differences. The residual standard deviation (RSD) is a measure of the random differences between the two methods. 95% of random differences are expected tol may nL ear m

ssiwt

rldtrriao

ie in the interval −1.96 RSD to +1.96 RSD. If this interval is large, the two methods

inear model validity: the Cusum test for linearity is used to evaluate how well a lin

urvey after consultation with CT technical and clinical staff. Theurvey was completed by a senior radiographer who specialisedn CT prior to collection of the PACS data, and the same individual

as consulted regarding matching of PACS examination codes tohe clinical scenarios used in the survey.

Since the study was conducted at a single hospital site, theesults may not represent concordance between the two data col-ection methodologies in different sites; thus it is unknown if theisparities observed would have been larger or smaller if mul-iple providers were included in the study. The study was alsoestricted in the number of CT examinations evaluated; however, a

ange of examination methodology and anatomical locations werencluded. This study was limited to an evaluation of reliability andssumed PACS data as the gold standard. Reliability is only onef the potential limitations to self-complete survey data collection

ot be comparable.odel fits the data.§Indicates significant deviation from linearity (i.e. P value <0.05).

instruments and issues such as response bias and response fractionwere not evaluated. In our study, due to the manual nature of thePACS data extraction, we required the provider to agree to partic-ipate in the study and hence there would have been no benefit interms of response fraction. However, if the recommendation of theEuropean Commission on Radiation Protection [30] is realised (i.e.that national authorities responsible for population dose surveysgather electronic information on patient doses from RIS/PACS intonational databases established for the development and monitor-ing of DRLs and population dose estimates) then this data collectionmethod would be far superior in terms of both response fraction

and response bias. Our study did not evaluate the cost differencesbetween the two methodologies; however, since collection of PACSdata for our study was done manually (because data were notable to be extracted automatically), the PACS data collection was

336 R.E. Moorin et al. / European Journal of Radiology 83 (2014) 329– 337

Fig. 1. Bland–Altman analysis of the agreement in effective dose (a) and organ radiation dose (b) using survey data compared with data extracted from departmental picturearchival communication system for all CT examination types evaluated.

Fig. 2. Bland–Altman analysis of the agreement in organ radiation dose produced using survey data compared with data extracted from departmental picture archivalcommunication system according to type of CT examination.

rnal o

svtlercgbecattutac

5

sdsafoieaantad

C

F

t(pp

R

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[30] European Commission Dose Datamed Project. European Guidance on Estimat-ing Population Doses from Medical X-Ray Procedures, Radiation Protection

R.E. Moorin et al. / European Jou

ignificantly more expensive to undertake compared with the sur-ey. This additional cost needs to be considered when evaluatinghe relative advantages and disadvantages of the two data col-ection methods. Should national or even local databases becomestablished, then the cost of such data would be significantlyeduced. Implementing dose databases for DRL and audit purposesan maintain the anonymity of patients by only collating or aggre-ating the data required for dose calculation. Such a system woulde complicit with Australian health data and biospecimen law [31],nsuring no breach of legal obligations is made and the ethical cal-ulus strongly favours the public interest in radiation protectionnd avoidance of unnecessary harm to patients. Data from bothhe survey and PACS data were analysed in an identical mannero produce organ and effective doses and this method has beensed previously and is well established. Analysis of the dosime-ry data for reliability was undertaken using the most appropriatedvanced statistical methods and provided a range of indicators ofoncordance.

. Conclusion

Due to increasing concerns about the radiation dose from CTcanning examinations, robust methods for collection of CT doseata are required. Currently self-report surveys are used exten-ively for this purpose but suffer from limitations. The increasingvailability of electronic databases to capture information requiredor the evaluation and monitoring of CT radiation dose provides anpportunity for better quality data in a cost-effective manner, elim-nating many of the potential biases. This study has confirmed dataxtracted from RIS/PACS is superior to self-reported survey datand has shown survey data contains both proportional and system-tic bias not consistent across CT examinations. We recommendational and local databases that are established to routinely cap-ure aggregated and anonymous CT dose data for the developmentnd monitoring of DRLs and surveillance of population radiationose.

onflict of interest

The authors of this paper have no conflict of interest to declare.

unding sources

Financial support for this study was provided by a grant fromhe National Medical and Research Council (Australia) project grantAPP 1008394). The funding agreement ensured the author’s inde-endence in designing the study, interpreting the data, writing andublishing the report.

eferences

[1] Golding SJ, Shrimpton PC. Radiation dose in CT: are we meeting the challenge.The British Journal of Radiology 2002;75:1–4.

[2] Naik KS, Ness LM, Bowker AMB, Robinson PJA. Is computed tomography of thebody overused? An audit of 2068 attendances in a large acute hospital. British

Journal of Radiology 1996;69:126–31.

[3] Hart D, Wall BF. UK population dose from medical X-ray examinations. Euro-pean Journal of Radiology 2004;50:285–91.

[4] Shrimpton PC, Wall BF, Hart D. Diagnostic medical exposures in the UK. AppliedRadiation and Isotopes 1999;50:261–9.

[

f Radiology 83 (2014) 329– 337 337

[5] Shrimpton PC, Wall BF. The increasing importance of x-ray computed tomo-graphy as a source of medical exposure. Radiation Protection Dosimetry1995;57:413–5.

[6] Aroua A, Burnand B, Vader JP, Valley JF. Nation-wide survey on radiationdoses in diagnostic and interventional radiology in Switzerland in 1998. HealthPhysics 2002;83:46–55.

[7] Kaul A, Bauer B, Bernhardt J, Nosske D, Veit R. Effective doses to members ofthe public from the diagnostic application of ionizing radiation in Germany.European Radiology 1997;7:1127–32.

[8] Brugmans MJP, Buijs WCAM, Geleijns J, Lembrechts J. Population exposureto diagnostic use of ionizing radiation in the Netherlands. Health Physics2002;82:500–9.

[9] Mettler FAJ, Wiest PW, Locken JA, Kelsey CA. CT scanning: patterns of use anddose. Journal of Radiological Protection 2000;20:353–9.

10] Rehani MMCT. Caution on radiation dose. Indian Journal of Radiology and Imag-ing 2000;10:19–20.

11] Wall BF. Implementation of DRLs in the UK. Radiation Protection Dosimetry2005;114:183–7.

12] Shrimpton PC, Hillier MC, Lewis MA, Dunn M. National survey of doses from CTin the UK:2003. The British Journal of Radiology 2006;79:968–80.

13] Pearce MS, Little MP, McHugh K, et al. Radiation exposure from CT scans inchildhood and subsequent risk of leukaemia and brain tumours: a retrospectivecohort study. Lancet 2012, http://dx.doi.org/10.1016/S0140-6736(12)60815-0.

14] Moorin R, Forsyth R, Gibson D, Fox R. Radiation dosimetry assessment of rou-tine CT scanning protocols used in Western Australia. Journal of RadiologicalProtection 2013;33:295–312.

15] Sodickson A, Warden GI, Farkas CE, et al. Automated anatomy-specific CTradiation exposure extraction for quality assurance and radiation monitoring.Radiology 2012;264:397–405.

16] Sodickson A. CT radiation risks coming into clearer focus. British Medical Jour-nal 2013;346:f3102.

17] Imaging and Performance and Assessment of CT scanners (ImPACT). CT patientdosimetry Excel spreadsheet. Imaging and Performance and Assessment ofCT scanners evaluation centre of the DH Medicines and Healthcare productsRegulatory Agency (MHRA); 2001 http://www.impactscan.org

18] Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability.Psychological Bulletin 1979;86:420–8.

19] McGraw KO, Wong SP. Forming inferences about some intraclass correlationcoefficients. Psychological Methods 1996;1:30–46.

20] Lin LI-K. A concordance correlation coefficient to evaluate reproducibility. Bio-metrics 1989;45:255–68.

21] McBride. A proposal for strength-of-agreement criteria for Lin’s ConcordanceCorrelation Coefficient. NIWA Client Report: HAM2005-062. Hamilton, NewZealand: National Institute of Water & Atmospharic Research Ltd; 2005.

22] Passing H, Bablok W. A new biometrical procedure for testing the equality ofmeasurements from two different analytical methods. Application of linearregression procedures for method comparison studies in Clinical Chemistry,Part I. Journal of Clinical Chemistry, Clinical Biochemistry 1983;21:709–20.

23] Bland JM, Altman DG. Measuring agreement in method comparison studies.Statistical Methods in Medical Research 1999;8:135–60.

24] Krouwer JS. Why Bland-Altman plots should use X, not (Y + X)/2 when X is areference method. Statistics in Medicine 2008;27:778–80.

25] Committee to Assess Health Risks from Exposure to Low Levels of IonizingRadiation. Health risks from exposure to low levels of ionizing radiation: BEIRVII-Phase 2. Board on Radiation Effects Research, Division on Earth and LifeStudies, National Research Council; 2006.

26] Mathews JD, Forsythe AV, Brady Z, et al. Cancer risk in 680 000 people exposedto computed tomography scans in childhood or adolescence: data linkage studyof 11 million Australians. British Medical Journal 2013;346:f2360.

27] Institute of Physics and Engineering in Medicine (IPEM). Guidance on theestablishments and use of diagnostic reference levels for medical X-ray exam-inations. IPEM Report No 88. York: Institute of Physics and Engineering inMedicine; 2004.

28] Friberg EG, Widmark A, Ryste Hauge IH. National collection of local diagnosticreference levels in Norway and their role in optimization of X-ray examinations.Osteras, Norway: Norwegian Radiation Protection Authority; 2008.

29] Kharuzhyk SA, Matskevich SA, Filjustin AE, Bogushevich EV, Ugolkova SA. Sur-vey of computed tomography doses and establishment of national diagnosticreference levels in the Republic of Belarus. Radiation Protection Dosimetry2010;139:367–70.

No.154. Chilton, Didcot, Oxfordshire: Health Protection Agency; 2008.31] Holman CDJ. Anonymity & research. Health data and biospecimen law in

Australia. Perth: Uniprint, The University of Western Australia; 2012.