QIBA CT VolumetricsQIBA CT Volumetrics Group 1B: Group 1B:

(Patient Image Datasets) (Patient Image Datasets)

ChargeCharge

1.1. To agree on questions to be answered To agree on questions to be answered by these Reference Datasetsby these Reference Datasets

2.2. To identify requirements for those To identify requirements for those datasets, based on questions to be datasets, based on questions to be answeredanswered

3.3. To identify existing datasets that can be To identify existing datasets that can be leveraged to provide desired datasetsleveraged to provide desired datasets

Questions (overview)Questions (overview)

1.1. What level of What level of accuracy and precisionaccuracy and precision can be achieved can be achieved in measuring tumor volumes in patient datasets?in measuring tumor volumes in patient datasets?

2.2. What level of reproducibility in estimating What level of reproducibility in estimating changechange can can be achieved when measuring tumors in be achieved when measuring tumors in phantomphantom datasets?datasets?

3.3. What is the What is the minimum detectable level of changeminimum detectable level of change that that can be achieved when measuring tumors in can be achieved when measuring tumors in patient patient datasets under a “No Change” conditiondatasets under a “No Change” condition??

4.4. What level of reproducibility in estimating What level of reproducibility in estimating changechange can can be achieved in measuring tumors in be achieved in measuring tumors in patient datasets patient datasets with “Unknown Change” conditionwith “Unknown Change” condition??

5.5. What is the effect of slice thickness on estimating What is the effect of slice thickness on estimating changechange in tumors using in tumors using patientpatient datasets? datasets?

Initially Agreed to First Pursue:Initially Agreed to First Pursue:

1.1. What level of What level of accuracy and precisionaccuracy and precision can be achieved can be achieved in measuring tumor volumes in patient datasets?in measuring tumor volumes in patient datasets?

2.2. What level of reproducibility in estimating What level of reproducibility in estimating changechange can can be achieved when measuring tumors in be achieved when measuring tumors in phantomphantom datasets?datasets?

3.3. What is the What is the minimum detectable level of changeminimum detectable level of change that that can be achieved when measuring tumors in can be achieved when measuring tumors in patient patient datasets under a “No Change” conditiondatasets under a “No Change” condition??

4.4. What level of reproducibility in estimating What level of reproducibility in estimating changechange can can be achieved in measuring tumors in be achieved in measuring tumors in patient datasets patient datasets with “Unknown Change” conditionwith “Unknown Change” condition??

5.5. What is the effect of slice thickness on estimating What is the effect of slice thickness on estimating changechange in tumors using in tumors using patientpatient datasets? datasets?

Next StepsNext Steps

Define requirements and Design Define requirements and Design Experiments for these two questionsExperiments for these two questions

Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)

in measuring tumor volumes in measuring tumor volumes 1.1. Specific AimsSpecific Aims

(a)(a) Investigate both bias and variance of both readers and Investigate both bias and variance of both readers and algorithm-assisted readers in measuring volumes, algorithm-assisted readers in measuring volumes, diameters and bi-directional diameters of lesionsdiameters and bi-directional diameters of lesions

(b)(b) Investigate inter-observer variability in each taskInvestigate inter-observer variability in each task

(c)(c) Investigate Intra-observer variability in each task ???Investigate Intra-observer variability in each task ???

(NOTE: here observer should be interpreted broadly – as (NOTE: here observer should be interpreted broadly – as reader measuring manually for diameters as well as reader measuring manually for diameters as well as algorithm-assisted reader measuring contours).algorithm-assisted reader measuring contours).

Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)

in measuring tumor volumes in measuring tumor volumes 1.1. METHODS and MATERIALSMETHODS and MATERIALS LIDC datasetLIDC dataset lesions with “known size” will be used (size lesions with “known size” will be used (size

based on contours from 4 LIDC readers)based on contours from 4 LIDC readers) Each lesion will have boundariesEach lesion will have boundaries Only a single time point is needed (no followup Only a single time point is needed (no followup

needed, no diagnosis needed)needed, no diagnosis needed) Need to identify which nodules to use – Need to identify which nodules to use –

criteria?criteria?

Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)

in measuring tumor volumes in measuring tumor volumes 1.1. METHODS and MATERIALSMETHODS and MATERIALS Like NIST Biochange 2008, lesions are Like NIST Biochange 2008, lesions are

identified and coordinates provided to readers.identified and coordinates provided to readers. Reader tasksReader tasks

Manually mark 2 Diameters (w/o LIDC marks)Manually mark 2 Diameters (w/o LIDC marks)• Longest diameter and perpendicular diameterLongest diameter and perpendicular diameter

Also semiautomated contour of lesionAlso semiautomated contour of lesion• From contour, determine volume, longest diameter and From contour, determine volume, longest diameter and

diameter perpendicular to longest diameterdiameter perpendicular to longest diameter Have readers perform some cases > 1 (intra reader Have readers perform some cases > 1 (intra reader

variation)variation)

Data CollectedData Collected

Nodule #Nodule # coordscoords LIDC LIDC volvol

LIDC LIDC Longest Longest DiamDiam

LIDC LIDC Perp Perp DiamDiam

R1 R1 volvol

R1 R1 LD LD from from contcontourour

R1 R1 PD PD from from contocontourur

R1 R1 LDLD

R1 R1 PDPD

R2 R2 volvol

Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)

in measuring tumor volumes in measuring tumor volumes 1.1. METHODS and MATERIALSMETHODS and MATERIALS AnalysesAnalyses

LIDC would be considered “truth”LIDC would be considered “truth” Estimate bias of each readerEstimate bias of each reader

• VolumeVolume• Diam (manual and Assisted)Diam (manual and Assisted)• Product of diameters (LD x PD) (Manual and assisted) Product of diameters (LD x PD) (Manual and assisted)

Estimate inter-reader variabilityEstimate inter-reader variability• Reader vs. “gold standard”Reader vs. “gold standard”• Reader vs. reader Reader vs. reader

Intra-reader variabilityIntra-reader variability

Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)

in measuring tumor volumes in measuring tumor volumes 1.1. QuestionsQuestions

1.1. How many cases?How many cases?2.2. How many readers?How many readers?3.3. Case composition (all spherical? Some Case composition (all spherical? Some

spiculated? Range of sizes?)spiculated? Range of sizes?)4.4. Enough of each subgroup to perform a stat. Enough of each subgroup to perform a stat.

sign analysis?sign analysis?1.1. (reader bias on spherical nodules is XX; (reader bias on spherical nodules is XX; 2.2. (reader bias on spiculated nodules is YY)(reader bias on spiculated nodules is YY)

5.5. Do we want to do both size and shape Do we want to do both size and shape subgroup analyses?subgroup analyses?

QuestionsQuestions

Question 2 – What is the min detectable level of Question 2 – What is the min detectable level of change in patient datasets under a “No Change” change in patient datasets under a “No Change”

condition?condition?

1.1. Specific AimsSpecific Aims(a)(a) For patient datasets acquired over a very short time interval For patient datasets acquired over a very short time interval

(presumably the “no change” condition”) investigate variance of (presumably the “no change” condition”) investigate variance of both readers and algorithm-assisted readers in measuring both readers and algorithm-assisted readers in measuring changechange in volume, diameter and bi-directional diameters of lesions (here, in volume, diameter and bi-directional diameters of lesions (here, the expected value of the the expected value of the changechange should be zero) should be zero)

(b)(b) Investigate several change metrics such as:Investigate several change metrics such as:(a)(a) Absolute value of changeAbsolute value of change(b)(b) fractional change in volume/diameterfractional change in volume/diameter(c)(c) Categorical variables (progression, regression, etc.)??Categorical variables (progression, regression, etc.)??

(c)(c) Investigate inter-observer variability in each taskInvestigate inter-observer variability in each task(d)(d) Investigate Intra-observer variability in each task ???Investigate Intra-observer variability in each task ???(NOTE: again, observer should be interpreted broadly – as reader (NOTE: again, observer should be interpreted broadly – as reader

measuring manually for diameters as well as algorithm-assisted measuring manually for diameters as well as algorithm-assisted reader measuring contours).reader measuring contours).

Existing ResourcesExisting Resources

RIDER – MSK Coffee Break Experiment RIDER – MSK Coffee Break Experiment (No Change Condition)(No Change Condition)

32 NSCLC patients32 NSCLC patients Imaged twice on the same scanner w/in 15 Imaged twice on the same scanner w/in 15

minutesminutes Thin section (1.25 mm) imagesThin section (1.25 mm) images Manual linear measurements performed by 3 Manual linear measurements performed by 3

readers; volume obtained from algorithm.readers; volume obtained from algorithm.

QuestionsQuestions

1.1. METHODS and MATERIALSMETHODS and MATERIALS MSK Coffee break experiment patient MSK Coffee break experiment patient

datasetsdatasets Repeat scan of lesions over short time interval Repeat scan of lesions over short time interval

--- “No change” condition--- “No change” condition True size is unknownTrue size is unknown

QuestionsQuestions

1.1. METHODS and MATERIALSMETHODS and MATERIALS Like NIST Biochange 2008, lesions are identified and Like NIST Biochange 2008, lesions are identified and

coordinates provided to readers.coordinates provided to readers. Reader tasksReader tasks

For each lesion, manually mark 2 Diameters For each lesion, manually mark 2 Diameters • Longest diameter and perpendicular diameterLongest diameter and perpendicular diameter

Also each reader obtains a semiautomated contour of entire Also each reader obtains a semiautomated contour of entire volume of lesionvolume of lesion

• From contour, determine volume, longest diameter and From contour, determine volume, longest diameter and diameter perpendicular to longest diameterdiameter perpendicular to longest diameter

• Keep actual contour coordinates/mask (TBD)Keep actual contour coordinates/mask (TBD) Randomize order of cases so reader does not simultaneously Randomize order of cases so reader does not simultaneously

review both scans of a patient (does not see both scans of same review both scans of a patient (does not see both scans of same lesion); has to contour them independently.lesion); has to contour them independently.

Data CollectedData Collected

QIBA QIBA Patient ID Patient ID ##

Scan#Scan# Nodule Nodule ##

coordscoords R1 R1 volvol

R1 R1 LD LD from from contourcontour

R1 R1 PD PD from from contourcontour

R1 R1 LDLDmanualmanual

R1 R1 PDPDmanualmanual

R2 R2 volvol

11 11 11 (320,267, (320,267, 52)52)

2525

11 22 11 (310, (310, 248, 48)248, 48)

2222

11 11 22 (221,335, (221,335, 13)13)

3434

11 22 22 (215, (215, 340, 15)340, 15)

2828

Question 1 – Accuracy and Question 1 – Accuracy and Precision (aka Bias and Variance) Precision (aka Bias and Variance)

in measuring tumor volumes in measuring tumor volumes 1.1. METHODS and MATERIALSMETHODS and MATERIALS AnalysesAnalyses

Estimate variance of different change metrics forEstimate variance of different change metrics for• VolumeVolume• Diam (manual and Assisted)Diam (manual and Assisted)• Product of diameters (LD x PD) (Manual and assisted) Product of diameters (LD x PD) (Manual and assisted)

Estimate inter-reader variabilityEstimate inter-reader variability

Intra-reader variability ?? (can we do that with data we Intra-reader variability ?? (can we do that with data we would have? Or would we have to specifically introduce would have? Or would we have to specifically introduce the exact same cases multiple times?)the exact same cases multiple times?)

Questions to be Pursued in a Questions to be Pursued in a Future PhaseFuture Phase

QuestionsQuestions

2. 2. What level of reproducibility in estimating What level of reproducibility in estimating changechange can be achieved can be achieved in measuring tumors in in measuring tumors in phantomphantom datasets? datasets?

(a)(a) RECIST change vs. volume changeRECIST change vs. volume change(b)(b) Investigate just variance (not bias)?Investigate just variance (not bias)?(c)(c) inter-observer variabilityinter-observer variability(d)(d) Intra-observer variabilityIntra-observer variability(e)(e) Change metric – absolute value? fractional change in volume/diameter? Change metric – absolute value? fractional change in volume/diameter?

categorical variable?categorical variable?

For this question:For this question: Variety of lesions with known sizeVariety of lesions with known size Compare “size” of different lesions somehow (TBD)Compare “size” of different lesions somehow (TBD)

Use different existing lesions and treat them as though they were the Use different existing lesions and treat them as though they were the same lesion at different time points.same lesion at different time points.

Physically alter lesions over time and scan at both time points (Bob Ford’s Physically alter lesions over time and scan at both time points (Bob Ford’s water balloon experiment)water balloon experiment)

QuestionsQuestions4. 4. What level of reproducibility in estimating What level of reproducibility in estimating changechange can be achieved in can be achieved in

measuring tumors in measuring tumors in patientpatient datasets datasets with “Unknown Change” conditionwith “Unknown Change” condition??(a)(a) RECIST change vs. volume changeRECIST change vs. volume change(b)(b) Investigate just variance?Investigate just variance?(c)(c) inter-observer variabilityinter-observer variability(d)(d) Intra-observer variability (would be good to have more than 2 time points; may not Intra-observer variability (would be good to have more than 2 time points; may not

exist in RIDER).exist in RIDER).(e)(e) Change metric – absolute value? fractional change in volume/diameter? categorical Change metric – absolute value? fractional change in volume/diameter? categorical

variable?variable?(f)(f) Look at effects of lesion size for a specific (thin) slice thicknessLook at effects of lesion size for a specific (thin) slice thickness

For this question:For this question: Variety of lesions (true size may be unknown)Variety of lesions (true size may be unknown) Patient datasets with same lesions at different time points.Patient datasets with same lesions at different time points. Lesions may or may not have changed size – change is unknown.Lesions may or may not have changed size – change is unknown. RIDER datasets – unannotated as of yet. We have identified about 20 RIDER datasets – unannotated as of yet. We have identified about 20

lesions of various sizes. Possible cases from RadPharmlesions of various sizes. Possible cases from RadPharm..

QuestionsQuestions5. 5. What is the effect of What is the effect of slice thicknessslice thickness on on estimating changeestimating change in tumors using in tumors using patientpatient

datasets?datasets?(a)(a) RECIST change vs. volume changeRECIST change vs. volume change(b)(b) Investigate just variance?Investigate just variance?(c)(c) inter-observer variabilityinter-observer variability(d)(d) Intra-observer variabilityIntra-observer variability(e)(e) Change metric – absolute value? fractional change in volume/diameter? categorical variable?Change metric – absolute value? fractional change in volume/diameter? categorical variable?(f)(f) Look at interactions between slice thickness and lesion size (and other lesion characteristics Look at interactions between slice thickness and lesion size (and other lesion characteristics

such as shape, margin, etc.)such as shape, margin, etc.)

For this question:For this question: Also Variety of lesions (true size may be unknown)Also Variety of lesions (true size may be unknown) Patient datasets with same lesions at different time points, all done originally with Patient datasets with same lesions at different time points, all done originally with

thin slices; create a thick slice series (average together adjacent images).thin slices; create a thick slice series (average together adjacent images). Lesions may or may not have changed size – change is unknown.Lesions may or may not have changed size – change is unknown. Estimate change on thin and thick series and compareEstimate change on thin and thick series and compare Thin Slice RIDER datasets that can be fused together. Other cases from Thin Slice RIDER datasets that can be fused together. Other cases from

RadPharm?RadPharm?