Some experiments using modality classification in a multimodal repository: Medical Collection of ImageCLEF 2011

A.Castellanos1, Ana Garca-Serrano1, X. Benavent2

1

Universidad Nacional de Educacin a Distancia, UNED 2 Universitat de Valncia

{acastellanos, agarcia}@lsi.uned.es, xaro.benavent@uv.es

One main aspect in retrieval systems for multimodal repositories within current research is the design of user interfaces in order to improve the effective communication between users and the multimodal system. In this work it is presented several experiments performed in order to study the impact in the search of available visual classification of the search results. Previous studies have shown that visual modality of images is relevant for the multimedia retrieval in domain repositories as the medical ones []. So, our main goal is to study the influence of visual classification of images in textual-based search results, to conclude about its integration or not in the interfaces of multimodal Information Retrieval (IR) systems. The start point of our work is our participation in the Medical Retrieval Task included in the 2011 international campaign for multimedia and cross-lingual evaluation ImageCLEF [], and more specifically in the ad-hoc image-based retrieval subtask: an expert poses a query to the IR system and expects to obtain the set of relevant images (user satisfaction). One of our experiments where the second best result in textual retrieval ones [], just behind the work of LABERINTO research group [], but the relevance of the results obtained taking into account the modality classification in the expansion of the user query wasnt enough significant to conclude about the inclusion of visual modality in the user interface for textual and mixed (textual and visual) search. So, we have developed different experiments in order to investigate the best way for carry out the expansion. The Medical Collection provided to the participants at ImageCLEF 2011 is composed by a subset of 230.088 images of PubMed Central (http://www.ncbi.nlm.nih.gov/pmc/). The collection is structured in medical articles, each including different number of images. The images are described by a textual caption (metadata are: url, figure iri and their caption). Along with the collection a modality classification of the images is provided. This classification has 18 different categories (3D: 3d reconstruction, AN: angiography, CM: compound figure (more than one type of image), CT: computed tomography, DM: dermatology, DR: drawing, EM: electronMicroscopy, EN: endoscopic imaging, FL: fluorescence, GL: gel, GX: graphs, GR: gross pathology, HX: histopathology, MR: magnetic resonance imaging, PX: general photo, RN: retinograph, US: ultrasound, XR: xray). Each image has associated a unique category (metadata field so-called tag).

Within the ImageCLEF experiments the query expansion is performed as follows:

1. In all of the runs the stopwords are removed and stemming is performed. 2. Remove domain stopwords (Run2 & Run 3): Terms relatives to image domain like photograph, view or images was removed in order to focus the query in medical concepts. The search is performed against the caption field of the images in the collection. 3. Inclusion of the modality classes (Run 5 & Run 6): For each query it is identify the results modality classification expected (if possible) and added to the original query. The search is performed against the caption field and the tags field (which includes the modality classification) of the images in the collection. The inclusion of the modality classification in Runs 5 and 6 is performed as follows: 1- Identification of the results expected modality classification. 2- Pre-processing of the original query: delete of domain stopwords and common stopwords (like in Run2) 3- Query expansion with the information on the tags field of the images. In Run 5 the field was aggregated (with OR) to the original query and in Run 6 the field was used like a filter (with AND):
Original query: chest CT images with emphysema. Run5: (tags:CT)^0.7 OR (chest CT AND emphysema)^0.3 Run6: (tags:CT) AND (chest CT AND emphysema)

Note that the information about modality classification is included for double, so his cause that the medical semantic of the query decreases.
Position 2 6 53 55 Name Run2 Run3 Run5 Run6 Type Without modality expansion Without modality expansion With modality expansion With modality expansion MAP 0.2158 0.2125 0.1309 0.1270 P@10 0.3533 0.3867 0.3433 0.3100 Relevant results 1383 1138 564 651

Table 1 UNED-UV ImageCLEF 2011 Global Results

In Table 1 are included the experiments (runs) sent to the campaign only based on the textual retrieval. The evaluation is based on three different measures: MAP, P@10 and number of relevant results retrieved. P@10: precision in the first ten results retrieved (number of relevant results in relation to the total). Number of relevant results retrieved (#)(useful to determine the recall). Mean of the average precision (MAP) for each query. Can be shown that, in this first steps, the inclusion of modality classification significantly worse the global results (MAP and number of relevant documents retrieved). The MAP measure shows the average of the precision and recall attained for all the queries, and the ranking of the results is also taken into account. So we need to investigate in more detail why these results are obtained, using a query-byquery analysis, that is taking into account just the queries (13 of 30) in which the modality identification and related modification of expansion of the queries has potential influence (the 13 queries with modality are shown at Table 3). With this assumption a RoadMap with different experiments has been performed. The general results are shown in Table 2 and detailed results for the queries are at table 3 as well as in figures 1, 2 and 3.

Run 1: This is the baseline run without modality (detailed results in fig. 2). The search is performed against caption field of the images.
Original query: chest CT images with emphysema. Run 1: chest emphysema.

Run 2: In this case modality information is not removed. The search is performed against caption field of the images.
Original query: chest CT images with emphysema. Run 2: chest CT emphysema.

In Table 2 is shown that this run improves the result of Run 1, both in terms of precision and recall. Figure 2 shows that results of queries with modality information (red dots in figure) are better than in Run 1. Run 3: This is similar to the previous run but modality provided by the user is expanded with the label of the modality classification.
Modality Classification: XR Label: x-ray Original query: chest CT images with emphysema. Run 3: chest (CT OR "computed tomography") AND emphysema

In the results (global and by query) can be seen that Run 3 improve slightly the Run 1 and obtains similar results to the Run 2 in terms of precision, but worse results taking into account the recall. Run 4: Similar to Run 3 but the modality in the original query is removed. The search it is performed against the tags field of the images in the collection, as well as to the caption of images as in Run3.
Original query: chest CT images with emphysema. Run 4: (tags: CT)^0.7 OR (chest emphysema)^0.3

Run 5: Like Run 4 but using the tags field like filter instead of aggregate it as in Run 4.
Original query: chest CT images with emphysema. Run 4: (tags: CT) AND (chest emphysema)

The general results shows that our assumption about modality was true and runs 4 y 5 significantly improves our results of ImageCLEF (see rows at Table 1 and Table 2). But still the results arent enough to achieve the performance of Runs 1, 2, and 3. One aspect that probably affects these results is that in this domain is necessary to have an accurate modality classification of the images in the collection as well as a good textual representation in the caption associated.
Name Run 1 Run 2 Run 3 Run 4 Run 5 MAP 0.2003 0.2158 0.2125 0.1883 0.1932 P@10 0.3333 0.3533 0.3867 0.3667 0.3793 Relevant results 1340 1383 1138 1078 988

Table 2 Global Results

Table 3, Figure 1, Figure 2 and Figure 3 shows a query-by-query detailed analysis. Only queries that have a modality classification associated are shown.

Each query is compared with the baseline run to see when the inclusion of modality classification improves the results and why. Some conclusions from the analysis: The inclusion of the modality (Run 2) generally improves the queries but the inclusion as a filter (Run 5) decrease the results (Table 2). Taking into account only Run 2, which improves the results, some queries retrieved worse or similar results than baseline run (marked in red bold and bold respectively in the Table 3). Some more details: Queries with microscopic modality (17 and 26) have similar results, indicating that this modality isnt useful for expansion. The queries not improving results (7, 13, 14 and 15) included implicitly the modality in the query, so the expansion includes redundant information (for instance in query #13, fractures are typical of x-ray images).
Query
Doppler ultrasound images (colored) chest CT images with emphysema sagittal views of head MRI images x-ray images of a tibia with a fracture medial meniscus MRI abdominal CT images showing liver blood vessels all x-ray images containing one or more fractures Angiograms containing the aorta CT images with a brain infarction microscopic pathology images of the kidney fetal MRI CT liver abscess CT or x-ray images showing the heart microscopic images of tissue from the cerebellum

Run 2
MAP decrease, caused by the increase of total documents retrieved. MAP increase but less relevant results MAP and relevant results increase MAP and relevant results decrease MAP increase MAP and relevant results increase Decrease of MAP and relevant results Similar result Decrease of MAP and relevant results Similar results MAP increase Large increase of MAP Large increase of MAP Similar or worse result

Run 5
Large decrease of MAP and relevant results. Similar MAP but much less relevant results MAP and relevant results decrease Large decrease of MAP and relevant results MAP decrease. No documents retrieved MAP and relevant results increase, but than in Run 2 Similar results but increase of MAP Large decrease of relevant results and MAP MAP increase but produced by the decrease of total results Large decrease of retrieved results and consequently MAP too Large decrease of retrieved results and consequently MAP too Large increase of MAP Increase of MAP Worse result

Table 3 - Results by Query comparison (respect baseline)

Figure 1 - Results of Run 1 by Query

Figure 2 - Results of Run 2 by Query

Figure 3 - Results of Run 5 by Query

It has been shown that modality query expansion improves the retrieval results as seen in global and query-by-query results so, is an issue to take into account during the design of user interface (recommending the inclusion of modality when known by the user), but the way to carry out the expansion has to be refined (for example not as a filter etc.) in future works. References
1. Castellanos, A., Benavent, X., Benavent, J.: UNED-UV at Medical Retrieval Task of ImageCLEF.

http://clef2011.org/resources/proceedings/Castellanos_Clef2011.pdf, CLEF 2011 working notes, Amsterdam, The Netherlands, 2011. 2. Jayashree Kalpathy-Cramer, Henning Mller, Steven Bedrick, Ivan Eggel, Alba Garcia Seco de Herrera, Theodora Tsikrika, "The CLEF 2011 medical image retrieval and classification tasks", http://www.clef2011.org/resources/proceedings/Overview_ImageCLEF_Medical_Retrieval_Clef20 11.pdf, CLEF 2011 working notes, Amsterdam, The Netherlands, 2011. 3. Mata, J., Crespo, M., Maa, M.J.: LABERINTO at ImageCLEF 2011 Medical Image Retrieval Task, CLEF 2011 working notes, Amsterdam, The Netherlands, 2011. 4. Tirilly, P., Lu, K., Mu, X., Zhao, T., Cao, Y.: On modality classification and its use in text-based image retrieval in medical databases. Content-Based Multimedia Indexing (CBMI), 2011 9th International Workshop on. pp. 109114. IEEE (2011).