Synchronized slice viewing of similar image series

Sharib Aliac , Antonio Foncubiertaa , Adrien Depeursingea , Fabrice Meriaudeauc , Osman Ratibd , Henning Mllerab u
a University b Medical

of Applied Sciences Western Switzerland (HESSO), Sierre, Switzerland; Informatics, University Hospitals & University of Geneva, Switzerland; c University of Burgundy, Le Creusot, France; d Nuclear Medicine, University Hospitals of Geneva, Switzerland
ABSTRACT

Comparing several the series of images is not always easy as the corresponding slices often need to be selected manually and in times where series contain an everincreasing number of slices this can mean manual work when moving several series to the corresponding slice. Particularly two situations were identied in this context: (1) patients with a large number of image series over time (such as cancer patients) frequently need to compare the series, for example to monitor tumor growth over time. Manually adapting two series is possible but with four or more series this can mean loosing time. Having automatically the closest slice by comparing visual similarity also in older series with diering slice thickness and inter slice distance can save time and synchronize the viewing instantly. (2) analyzing visually similar image series of several patients can prot from being viewed in a synchronized way to compare the cases, so when sliding through the slices in one volume the corresponding slices in the other volumes are shown. This application could be employed after contentbased 3D image retrieval has found similar series. Synchronized viewing can help nding the most relevant cases quickly. To allow for synchronized viewing of several image volumes, the target image series are registered and the existing slices are located in the volumes. Subsequently, the user can move inside the slides of one volume and select target volumes to be synchronized with. These volumes are then synchronized in that the slice closest to the original volume is shown even when the slice thickness or inter slice distances dier, and this fully automatically by comparing the visual image content. The system uses ITK (Insight toolkit) and VTK (visualization toolkit) for the image processing and analysis parts. A simple interface using Qt has been developed and an integration into the image viewer OsiriX has been started to make tbe tool available to a larger public. A rst evaluation with image volumes of the same slice thickness and inter-slice distance for lung CTs and brain MRIs shows a very good performance. Goal is now to increase the complexity of the task progressively to varying scanners and protocols and then to partial volumes and potentially to localize single slices in volumes. To allow for synchronized viewing, the target series are registered to the reference series and the slices are located in the volumes. Currently, the system uses 3D rigid registration and nonrigid methods are under investigation. Subsequently, the user can move inside the slices of one volume and all others are synchronized automatically also when slice thickness or inter slice distance vary. The system uses ITK and VTK for the image processing and visualization parts. A simple interface using Qt was developed and integration into OsiriX was started to make the tool available to a larger public and with a better user interface. A rst evaluation with lung CTs and brain MRIs shows a good performance for series of basically the same anatomic area. The goal is to increase the complexity of the task progressively to varying protocols and partial volumes and localize single slices in volumes. Keywords: medical image retrieval, information retrieval, synchronized viewing, 3D data analysis
Further author information: (Send correspondence to Henning Mller, Email: henning.mueller@hevs.ch) u

1. DESCRIPTION OF PURPOSE
Medical image retrieval has been an extremely active domain in the medical imaging eld over the past years.?, ? Image retrieval can supply similar images or cases with similar images to clinicians in case of doubt that can then be used to aid diagnosis as the past cases often have a detailed description including a conrmed diagnosis. When data sets include 3D data such as tomographic series,? it is often not trivial to quickly choose the most relevant cases and to compare the images, something that is usually fast in 2D. To synchronize the viewing of several 3D series mainly in axial views automatically, corresponding structures in the series have to be matched and then all slices in the series need to be located in the volumes. Information on slice thickness and inter slice distance can also be found in the DICOM header. For the synchronization, simple registration algorithms can be used assuming that the reference and target volumes have several structures in common and are indeed of the same area. Tools for registration such as ITK (Insight Toolkit) and MATLAB exist for these tasks. Applications of such synchronized viewing can be manifold as they can be used in any situation where several tomographic volumes need to be compared. One situation could be the case of patients with many followup image series, where the visual content in these series needs to be compared over time. In this case a clinician can scroll through the slices and then compare all target volumes, always having the multiple views synchronized automatically. For contentbased 3D image retrieval this can also help to better judge the results, particularly when the texture is the distinguishing element and not the shape of objects, which could be shown in 3Drendered views after segmentation. Manually adapting the slices can be an option when there are few series and few images per series but with modern scanners producing hundreds of images even for small areas, new tools become essential to deal with the increasing amount of data and give clinicians the possibility to analyze data eciently.

2. METHODS
Several existing tools were adapted for the task described in this paper. The Insight toolkit (ITK ) contains many tools for image processing and image registration. In this article ITK is used for the image processing algorithms such as the registration. VTK (Visualization Toolkit ) on the other hand allows visualization of data sets such as 3D volumes. For the quick development of the user interface the Qt framework was employed and the ITK/VTK code was integrated. Two sets of images have been used in a rst step. A database of 100 patients with interstitial lung diseases containing high resolution CT of the lung, mostly with 1mm slice thickness and 10mm inter slice distance was used.? A database of brain MRI images is also used to show the tools work for a variety of modalities.

3. RESULTS 3.1. Synchronizing

To synchronize several image volumes (mainly axial views) with a reference volume that is selected, all target volumes are automatically registered with the reference volume using 3D rigid registration of ITK. After the registration, the slices can be situated in the volumes and thus the closest slices to any given slice in the reference volume can in any situation be calculated and shown on screen, based on the registration and also based on the information of slice thickness and inter slice distance in the DICOM header. Visual similarity of the slices can also be used to synchronize the slices in an optimal manner. The current 3D rigid registration consists of rigid transformations that are a combination of rotations R and translations t: I2 = RI1 + t,

(1)

http://www.itk.org/ http://www.vtk.org/ http://qt.nokia.com/ The MR brain data sets and their manual segmentations were provided by the Center for Morphometric Analysis at Massachusetts General Hospital and are available at http://www.cma.mgh.harvard.edu/ibsr/.

Figure 1. SSD values for various registration tasks. The registration of the lung CT scans did not converge.

Figure 2. A rst screenshot of the test interface showing four image series at the bottom of the interface synchronized with a source series that is shown in an axial view and also in a simple 3D reconstruction.

where I1 and I2 are the intensity values of the voxels of the reference and registered images respectively. The transformations are applied to minimize the sum of squared dierences (SSD) between intensity values:? SSD = 1 Nx Ny Nz
Nx Ny Nz

2 I1 (i, j, k) I2 (x + i, y + j, z + k) .

(2)

i=1 j=1 k=1

Rigid transforms based on (1) are iteratively applied until the dierence of SSD between two iterations is minimized to a threshold q = 104 . The evolutions of SSD of consecutive iterations of the rigid registration are shown in Figure 1 for the database of lung CT images and brain MRIs. The synchronization of the series is rejected if SSD cannot achieve the quality threshold q, leading to a nonconvergence of the rigid registration. Other similarity measures that are invariant to greyscale value ranges are currently under investigation, such as the risk of nonconvergence proposed by Skerl et al.? 3D rigid registration delivered satisfying qualitative results for series of basically the same anatomic area and having sucient resolution in the z axis. If there is only a partial overlap between the reference and target volumes other techniques are required that are currently being implemented and tested. The current system uses a desktop PC with 16 GB of RAM and a 4core processor. Registering the series can be fast but may take quite long if rotations and translations are necessary. Depending on the number of iterations required to obtain convergence, the registration in the tests took between 3 seconds for series with basically no translation and rotation to almost 10 minutes for a series with very strong shift and rotation. So far, computation eciency was not a major goal. Synchronization is then fast, taking around 1 second per series in average. So far, contentbased retrieval of similar image series has only been tested with lung CT images using the methods described in.?

3.2. User interface

A simple test interface was developed using the Qt crossplatform application framework and then integrating the image processing part for the corresponding volumes. Figure 2 shows the interface with the view of the reference series including a 3D view and the target volumes that are synchronized based on a user manipulating the source series. When the user slides through the source series, the target series are all synchronized. The same tools can be used in a contentbased retrieval setting where several results series need to be synchronized but also for patients where several image series need to be compared over time. Such synchronized viewing can save time when handling several image series. Besides the shown test interface, the development of a plugin to OsiriX has started as this will make the tools available to a large number of radiologists with a better interface, thus potentially increasing the impact of the tool and obtaining more feedback on adapting the system.

4. NEW OR BREAKTHROUGH RESULTS

To the best of our knowledge this is the rst article describing a system that uses purely visual image analysis methods and registration to allow for synchronized viewing of several tomographic volumes of imaging data in a fully automatic way. Such automatic synchronization of several image series adapts the viewed slice of all

http://www.osirix-viewer.com/

volumes to the same place inside the volume without manual intervention making such viewing much faster and easier to use. This synchronized viewing can help working with patients containing many volumes of the same area that needs to be compared manually. For 3D contentbased medical image retrieval this can also help comparing several results series in a synchronized way, making it much faster to nd relevant data. Currently, synchronization requires to at least align one slice between several volumes, as then the slice thickness and inter slice distance of the DICOM headers can be used for making the correspondence.

5. CONCLUSIONS
The system developed in the project described in this paper has several practical applications and can make image viewing and image comparisons of 3D data sets much faster and more eective by synchronizing several volumes. The tool is based on open source components and thus allows to be reproduced fairly easily and without major cost. An integration into OsiriX will give access to the tools for a larger number of radiologists and will help a largescale validation of the tool. Currently, the validation has only been performed on data sets taken under fairly similar conditions, so with almost the exact same location of the volumes. Besides the advantages of the described tools we have also identied several shortcomings of the current system that are currently being worked on. It would be optimal to have standard models for all anatomic regions that all other volumes or even single slices can be mapped to. Single slices would be hard to register as only very limited information is available and nding the visually most similar slice does not necessarily mean to be in the same region. This could help extracting information from many information sources such as the medical literature (JPG images of limited resolution) and link these with certain regions and even local abnormalities in clinical cases in DICOM format, potentially connecting a clinical record with related articles in the medical literature.

6. SUBMITTED ELSEWHERE
The content of this paper has never been submitted elsewhere and is not planned to be submitted elsewhere.

7. ACKNOWLEDGEMENTS
We would like to thank the EU for their funding in the context of the Khresmoi project (grant 257528) and the Swiss National Science foundation (205321130046).