VOLCANO Studies:

Establishing a public benchmark for

evaluating change-in-size measurement
of pulmonary Nodules

Anthony P. Reeves
Vision and Image Analysis Group
School of Electrical and Computer Engineering
Cornell University

Presentation to the Lung Cancer Workshop VII, May 2010

 VOLCANO Concept

1. Develop a benchmark set of image pairs for characterizing the

performance of change-in-size lung nodule measurements

2. Establish benchmark results for the change in size measurements

using a variety of CAD methods

3. Establish benchmark results for manual measurements

4. Explore metrics to evaluate new measurement methods

5. Make the volcano dataset and results available to the public

6. Improve the benchmarks on the basis of the results and new

understanding

VIA
 Nodule Measurement Studies

VOLCANO: VOLume Change Analysis of Nodules

1. VOLCANO ‘09: Compare different automated computer

methods for measuring change in size of pulmonary
nodules.

2. VOLCAMAN ‘10: VOLCAno MANual Compare different

manual markings for measuring change in size of
pulmonary nodules.

• Develop a standardized evaluation method and data set

(Benchmark) for change-in-size measurement

VIA
 Outline

1. VOLCANO studies overview

2. VOLCANO ‘09 results
3. VOLCAMAN ‘10

• VOLCANO Goals
• A Benchmark set if image pairs
• Results from automated and manual evaluation
• Evaluation methodology

VIA
 Relative Change in Size

• The relative change in size of a lesion is a clinically

relevant measurement of pulmonary nodules
– To determine malignancy status in small nodules
– To determine the response to therapy
– To improve the efficiency of drug trials

• It is imperative to understand the precision in a change-

in-size measurement

VIA
 Volume or size measurement

• Most studies on lesion evaluation methods have been

based on volume or size measurement
• High accuracy in a volume measurement is not clinically
useful
• Accuracy in volume measurement is not useful for
predicting the accuracy and precision of change-in-size
measurements

• VOLCANO’09 did not require participants to submit

volume estimates.
– Some size change methods do not involve volume estimates
(12/17 methods did provide volume estimates)

VIA
 Ground Truth Challenge
A Challenge for this Challenge
There is a large amount of disagreement in experienced chest
radiologist estimates of pulmonary nodule volumes
– Lung Image Database Consortium (LIDC)1: 10 mm nodule limits of
agreement: (-23.4%, 27.7%)

• VOLCANO’09 compared computer methods with each other

instead of using expert radiologist volume estimates as ground
truth

1. A. P. Reeves, A. M. Biancardi, T. V. Apanasovich, C. R. Meyer, H. MacMahon, E. J.R.

van Beek, E. A. Kazerooni, D. Yankelevitz, M. F. McNitt-Gray, G. McLennan, S. G. Armato
III, C. I. Henschke, D. R. Aberle, B. Y. Croft, L. P. Clarke, The lung image database
consortium (LIDC): A comparison of different size metrics for pulmonary nodule
measurements. Academic Radiology, 14(12):1475-1485, Dec 2007.
 Volcano ‘09 Challenge Challenges

1. Volunteer activity

2. Data available from Prevent Cancer Foundation Database

No other publically available suitable data at that time

1. Volunteer participation
Participation incentives:
1. Authorship on main publication
2. Limited effort required
3. Individual feedback on performance (all published results are
anonymized), bragging rights may be claimed
4. Herding cats

VIA
Colima’s volcano in Colima, Mexico
 Overview

• The Volcano’09 Dataset

• The Analysis Methods
• The Challengers
• The Results
 Data Set Design Considerations

1. Data set size: The data set must not be too large as to require to much
time to evaluate:
50 image pairs for CAD, 24 image pairs for Manual

2. Nodule complexity: Given the size constraint nodules without

excessive complexity issues were selected.

3. Degree of size change:

clinical relevance is the smallest change that can be reliably
measured. Most cases have from 0 to less than 50% size change

4. CT Image quality:
All scans are low-dose, high-resolution, non-contrast, whole-lung view
with thin slices (1.25 mm for most scans)
-- A major consideration was the data available to the study

VIA
 Data Selection: Four Data Subsets

• Data subsets with specific properties were selected to address

different measurement issues
• Participants were not informed in advance of the different data
subsets
Group # cases Description
A 14 Zero-change cases with same slice thickness
- Intrinsic variation in size measurements
B 13 Zero-change cases with different slice thickness
- Measurement sensitivity to slice thickness
C 22 Cases with actual change (3 with large change)
- Real size change cases
D 1 Phantom with different slice thickness
- Known volume

VIA
 Nodule Size Distribution

12
mm3
5.8 5.8 – 7.3 7.3 – 9.1 9.1 – 11.5 11.5 – 14.5 14.5 – 18.3 18.3 mm

Distribution of nodule sizes, expressed in volume (mm3), separated according to group

Mount Fuji, Japan

The VOLCANO’09 Challenge: Preliminary Results

A. P. Reeves, A. C. Jirapatnakul, A. M. Biancardi, T. V. Apanasovich, C.
Schaefer, J. J. Bowden, M. Kietzmann, R. Korn, M. Dillmann, Q. Li, J. Wang, J.
H. Moltz, J. Kuhnigk, T. Hayashi, X. Zhou, H. Fujita, T. Duindam, B. van
Ginneken, R. Avila, J. P. Ko, K. Melamud, H. Rusinek, R. Wiemker, G. Soza, C.
Tietjen, M. Thorn, M. F. McNitt-Gray, Y. Valenciaga, M. Khatonabadi, Y. Kawata,
and N. Niki. "The VOLCANO'09 challenge: Preliminary results," In Second
International Workshop of Pulmonary Image Analysis, pp. 353-364, Sept. 2009
 Acknowledgements

• David F. Yankelevitz (Weill Cornell Medical College) for providing

and reviewing the cases used in this study
• Tatiyana V. Apanasovich (Thomas Jefferson University) for aiding in
the statistical analysis
• Lisa Kinnard (FDA) for providing the phantom nodule for this study
• Alberto Biancardi and Artit Jirapatnakul (VIA group)
For doing all the real work
• Brad Keller (VIA group) Volcano graphics

VIA
The VOLCANO Challenge Data Set
 Methods: Dataset Preparation

• Two scans for each nodule

• Patient information was removed from scans
• The order of the scans was randomized and scan dates changed to reflect 100
day interval between scans
• Scans were clipped in the axial direction, with five slices above and below the
nodule region (if possible)
• Seed point was provided for each nodule

VIA
Examples of Nodules

VIA
 Examples: No Change

Scan 1 Scan 2 3D Visualization (axial view)

Scan 1 Scan 2 3D Visualization (axial view)

VIA
 Example: Moderate Change
(~27%)

Scan 1 Scan 2

3D Visualization (axial view)

VIA
 Example: Different Slice Thickness

Scan 1 (1.25 mm) Scan 2 (2.5 mm)

3D Visualization
(axial view)

VIA
 Example: Large Change Nodule

Scan 1 Scan 2

3D Visualization
(axial view)

VIA
The Challengers
 Participants: 13 Teams
Team Members Team Members
Biomedsys Chris Schaefer, Jeffrey Kitware Rick Avila
J. Bowden New York Jane P. Ko, Kira Melamud,
Definiens Markus Kietzmann, University Henry Rusinek
Rene Korn, Markus Philips Rafael Wiemker
Dillmann
Siemens AG Grzegorz Soza, Christian
Duke Qiang Li, Jiahui Wang Tietjen, Matthias Thorn
Fraunhofer Jan H. Moltz, Jan- UCLA Michael F. McNitt Gray,
MEVIS Martin Kuhnigk Yanisley Valenciaga,
Gifu Tatsuro Hayashi, Maryam Khatonabadi
Xiangrong Zhou, University of Yoshiki Kawata, Noboru
Hiroshi Fujita Tokushima Niki
Image Thomas Duindam,
Sciences Bram van Ginneken VIA Cornell Anthony. P. Reeves, Artit.
Institute C. Jirapatnakul, Alberto. M.
Biancardi
 Submitted Methods:
Automation Level
Methods were ranked according to the following scale:

1. Totally automatic using seed points

2. Limited parameter adjustment ( < 15% of the cases)

3. Moderate parameter adjustment ( < 50% of the cases)
4. Extensive parameter adjustment ( ≥ 50% of the cases)

5. Limited image/boundary modification ( < 15% of the cases)

6. Moderate image/boundary modification ( < 50% of the cases)
7. Extensive image/boundary modification ( ≥ 50% of the cases)

VIA
 The Challengers (17 methods)

Team/Method Auto Method Team/Method Auto Method

Duke 1 Spiral scanning, ISI, SPHERE 4 Sphere fitting
dynamic ISI, SEG 4 Image filtering
programming
ISI, REG 4 Elastic registration
Gifu 1 Image filtering Kitware 4 Fast marching and
shape detection level
Philips 1 Active contour set
VIA, GAD 1 Density change VIA, GAS 4 Image filtering
Biomedsys 2 Image filtering Definiens 5 Context-based
MeVIS 3 Image filtering segmentation
and PVC UCLA 6 Image filtering
Siemens 3 Image filtering NYU, HYB 7 Image filtering and
and PVC PVC
Tokushima 3 Image filtering NYU, HYBA 7 Image filtering and
PVC
The Analysis Methods
 Analysis Methods

• Teams were required to provide a size change metric for

each nodule, e.g. for methods using volumes:
V2 − V1
RVC =
V1
• Non-parametric tests were used
• Variation was quantified using median absolute deviation
(MAD)
– Compute the median size change metric for each nodule, and
compute the absolute deviation from this median for each method
• Bias was quantified by median of absolute median (MAM)
– Compute median of the absolute median of size change metric for
each nodule

VIA
Results
Results: Group A
zero-change, same slice thickness

Nodule
 Example: Group A with large
median size change

Region of Scan 1 Scan 2

Interest

3D
Visualization
(axial view)
 Example: Group A with large
median size change

Region of Scan 1 Scan 2

Interest

3D
Visualization
(axial view)
Results: Group C
Nodules with actual change – small

Nodule Two outliers not shown

Results: Group C
(nodules with large change)
 Results: Group B
(zero-change, different slice thickness)

Nodule Presentation Order: thin-slice 1, thicker-slice 2

Results: Phantom

First scan (S1)

acquired at 0.7
mm slice
thickness; second
scan (S2) at 1.5
mm slice thickness
 CAD Method Evaluation
(Which CAD is best?)
Methods were evaluated on the following four metrics:

“Meanest of the mean”

(actually the medianest of the medians)
The method that best predicts the “mean” result

Best nothing
The method that estimates the smallest change for zero change

Slice master
The method with the least bias with respect to slice thickness

The Absolute in volume

The best volume estimation (for methods with volume data)

VIA not eligible to compete… but we would have been a contender

Meanest of the Mean
 http://www.via.cornell.edu/volcaman
• Study the manual volumetric measurement of lesions
• Participants manually mark nodule boundaries over the web
• Open to all physicians to self enroll

SIMBA VIA
VIA
 ‘10

Participants incentives

• What each participant is expected to do:

– Mark the boundaries of 24 nodule pairs over the web

• What they will get for it:

– A personalized anonymous evaluation of their performance with
respect to other participants
– An invitation to participate as an author in the VOLCAMAN
publication

SIMBA VIA
VIA
 VOLCAMAN Study Specifications - 1

• All activities are managed through the web

– Participants may access the study through any web connected computer that has a
display with sufficient resolution and quality.

– Participants may do the study in multiple sessions.

– Participants may take as much time to mark a nodule as they want; however, the
marking time is recorded as a study item.

– Participants should either complete the marking of the current nodule or cancel the
marking of the current nodule before ending a session.

SIMBA VIA
VIA
 VOLCAMAN Study Specifications - 2

• Stages of study participation (web managed)

1. Participants self register

2. Participants do the study training
3. QA: Organizers review registration and training results
4. On successful training, participants do the study
5. QA: Organizers review the study markings and request that
participants re-mark any cases where a problem is identified

SIMBA VIA
VIA
 VOLCAMAN Study Specifications - 3

• The Nodule marking task

1. Participants are presented with a “baseline” nodule and mark the

boundary on all image slices

2. Participants are then presented with a “follow up” nodule and

mark the boundary on all images. The baseline nodule and its
marking are available for side by side review for this process;
however, it is not possible to change the baseline marking.

SIMBA VIA
VIA
 VOLCAMAN Study Web-based marking

A Baseline marking tool B Follow up marking tool

Baseline image
Baseline image to with marks Follow up image
be marked to be marked

Image tasks A Baseline, B Follow up SIMBA VIA

VIA 5
 VOLCAMAN Study Specifications - 4

• The Data Set

– 24 of the image pairs used in the VOLCANO study

• Randomization
– The same nodule pair ordering will be used as for VOLCANO
– The order of the nodule cases for marking will be individually
randomized for each participant

SIMBA VIA
VIA
 Volcano Progress Report

1. Develop a benchmark set of image pairs for characterizing the

performance of change-in-size lung nodule measurements
Initial data set of 50 image pairs established
2. Establish benchmark results for the change in size measurements
using a range of CAD methods: VOCLANO ‘09
13 CAD teams provided results using 18 CAD methods
3. Establish benchmark results for manual measurements
Web-based VOLCAMAN ‘10 to provide manual results on 24
image pairs
4. Explore metrics to evaluate new measurement methods
5. Make the volcano dataset and results available to the public
VOLCANO ‘09 is still open and available to the public
6. Improve the benchmarks on the basis of the results and new
understanding

VIA