Sibgrapi 2010 : Tutorial submission

Graph-Based Image Segmentation

(Advanced tutorial)

Alexandre Falc
ao, Thiago Spina, Paulo Miranda, F
abio

Cappabianco
Institute of Computing - UNICAMP, Foundation Hermnio
Ometto - Uniararas
http://www.ic.unicamp.br/afalcao/

Abstract
The analysis of digital scenes often requires the segmentation of connected components,
named objects, in images and videos. The problem consists of defining the whereabouts of a
desired object (recognition) and its spatial extension in the image (delineation). Humans can
outperform computers in recognition, but the other way around is valid for delineation. The
interpretation of an image as a graph, whose nodes are the image pixels and arcs are defined by
some adjacency relation, provides different topologies to exploit optimum connectivity between
pixels for effective delineation, taking also advantage of a solid mathematical background from
graph theory. Recognition can be done by a human operator or object model. We will first
present a graph-based methodology, called Image Foresting Transform (IFT), which has been
used to develop fast and accurate delineation approaches. Interactive IFT-based methods exploit
a synergism between the computer for delineation and the user for recognition by simple marker
selection. Some of these methods will be explained and illustrated for image editing of natural
scenes using a common framework for arc-weight estimation. In the sequence, we will show
the links between image segmentation based on the IFT algorithm and the max-flow/mincut algorithm. IFT-based segmentation methods can also use object models or some other
prior knowledge about the problem to eliminate the user. We will then present a recent object
model, called cloud system, which combines shape information and IFT-based delineation in a
synergistic way. The cloud system model will be demonstrated for 3D automatic segmentation
of brain structures. Finally, we will conclude the tutorial by presenting an automatic brain
tissue classification method, which combines prior knowledge about the problem with IFTbased clustering.

Keywords
Graph-based image segmentation. Image Foresting Transform. Graph cut algorithms.
Differential IFT. Cloud models. Optimum-path forest clustering and classification.

Motivation
The segmentation of objects in digital images is crucial for the semantic analysis performed
by areas such as content based image retrieval (CBIR), natural image/video editing, and medical
image processing. In CBIR, descriptors can be extracted from the segmented object for the
retrieval of images from a database, whose content presents similar shapes for instance. The
entertainment and propaganda industries have intensively used foreground segmentation for
alpha matting of natural images and videos. Medical image processing needs a clear separation of
tissues and organs in 3D volumes obtained from MRI and CT devices for quantitative analysis.
Therefore, a great amount of research has been done to the development of efficient and effective
methods for interactive and automatic image segmentation.
The problem consists of defining the whereabouts of a desired object (recognition) and its
spatial extension in the image (delineation). Humans can outperform computers in recognition,
but the other way around is valid for delineation. In this sense, interactive methods, that exploit
the user for recognition and the computer for delineation in a sinergistic way, are desirable for
the sake of effectiveness in segmentation. In the automatic case, however, the user is usually
substituted by an object model or some other prior knowledge about the problem.
Most segmentation methods use some direct/indirect concept of arc weights between
pixels [11]. The weight may represent different functionals such as similarity, speed function,
affinity, cost, and distance; depending on different frameworks used, such as watershed, level
sets, fuzzy connectedness, and graph cuts. Indeed, the interpretation of an image as a graph,
whose nodes are the image pixels and arcs are defined by some adjacency relation, provides
different topologies to exploit optimum connectivity between pixels for effective delineation.
Besides, it is possible to take advantage of a solid mathematical background from graph theory.
In this tutorial, we will focus on a graph-based methodology, called Image Foresting
Transform (IFT) [4], and its most important and recent results for fast and accurate delineation.
In the IFT framework, interactive methods follow the aforementioned synergism between human
operator, who performs recognition by simple marker selection, and computer, which performs
IFT-based delineation (Figures 1a-1d). This synergism is also maintained for automatic
segmentation, using a novel model called object cloud system, which combines shape information
and IFT-based delineation. The tutorial will be prepared for an audience with some background
in image processing, pattern recognition, computer vision, data structures and graph-search
algorithms.
We will provide the basic concepts related to the IFT, its algorithm, implementation issues,
and variants, with emphasis on image segmentation in a first module of 50 minutes. Interactive
segmentation methods will be described in a second module of 40 minutes [12, 11, 3, 5],
using software demonstration and videos with an object tracking application based on IFT
segmentation and optical flow [6]. A third module of 60 minutes will show the links between the
IFT algorithm and the popular min-cut/max-flow algorithm [1], demonstrating the advantages
of the IFT-based approaches [10]. This module will also introduce the cloud system model
and its application to 3D automatic segmentation of brain structures from MR images [8, 9]

Motivation

(a)

(b)

(c)

(d)

Figure 1: (a-d) A sequence of user interventions for final segmentation, where the user adds
markers for recognition while the computer displays the results of delineation in real time.
(Figure 2). These images usually require inhomogeneity correction inside the brain in order to
classify the brain tissues for analysis. These topics will be covered in a fourth module of 30
minutes, where the automatic voxel classification is performed by an IFT-based clustering and
some prior knowledge about the problem [2]. A software, called Brain Image Analyzer (BIA),
will be used to demonstrate the results of the two last modules (see Figure 3).

Figure 2: (a) A coronal slice of a 3D cloud system with three objects (the cerebellum, and the
right and left hemispheres). (b) The recognition score for all positions and (c-d) the final 3D
segmentation of multiple objects.

Figure 3: A screenshot of the software BIA showing the results of 3D automatic segmentation
of brain tissues and structures.

Summary
The tutorial is organized in four modules comprising a total of 3 hours:
1. First module (50 min):
Introduction to image segmentation and graph-based approaches.
Adjacency relations and image-graphs.
Image Foresting Transform (IFT).
Connectivity functions.
General algorithm.
Implementation issues.
Important variants for graph-based image segmentation.
2. Second Module (40 min):
A common framework for arc-weight estimation in natural scenes.
Multiscale color attributes.
IFT-based fuzzy pixel classification.
Arc-weight estimation from image and object information.
Boundary-based segmentation using live-wire-on-the-fly.
Region-based segmentation using IFT with seed competition (IFTSC).
Software demonstrations and object tracking in video.
3. Third module (60 min):
The IFT algorithm versus the min-cut/max-flow algorithm for segmentation.
Cloud system model.
Definition and construction from multiple segmentations.
Object search strategies.
Automatic 3D MR-image segmentation of brain structures using cloud systems and
IFTSC for delineation.
Image-graph and arc-weight estimation.
Variants, software demonstration and results.
4. Fourth module (30 min):
Inhomogeneity correction in MR images of the brain.
An IFT-based clustering approach.
Automatic MR-image classification of brain tissues using IFT clustering and prior
knowledge about the problem.
Software demonstration.

First Module
The first module will introduce the image segmentation problem and the advantages
of interpreting an image as a graph. The IFT framework [4] will be presented by showing
that different adjacency relations and connectivity functions can lead to boundary-based [3]
and region-based [5, 2] approaches for segmentation. The IFT algorithm essentially reduces
the problem to an optimum-path forest computation in the image-graph. It will be presented
and its variants and implementation issues will be discussed. The module will conclude with
an introduction of the remaining modules, wherein interactive [12, 5] and automatic [8, 9, 2]
graph-based segmentation approaches will be demonstrated.

Summary

Second Module
Effective graph-based image segmentation strongly depends on suitable arc-weight estimation [11]. This module deals with interactive segmentation of natural scenes, and it will first
introduce how the arc weights can be better computed from the combination of a weight based
on multiscale color attributes and a weight based on object information. Object information is
obtained through fuzzy pixel classification using the IFT framework [12]. Two approaches for
boundary-based [3] and region-based segmentation [5] will be presented with software demonstration, wherein user recognition is reduced to simple seed selection, and their application to
object tracking in video.

Third Module
This module will introduce the graph cut (GC) segmentation as proposed in [1] and the
procedures adopted to circumvent its existing bias due to the use of the min-cut/max-flow
algorithm. It will be shown that the best result of GC segmentation is the one obtained by
IFTSC using the same seeds and arc weights. After that, we will present a novel solution,
called cloud system model (CSM) [8, 9], for automatic segmentation. CSM employs recognition
and delineation by IFT in a tightly coupled manner until an optimum state is reached. We will
present the definition and construction of CSM from multiple interactive segmentations, and
discuss search strategies using it. The module concludes by demonstrating the 3D automatic
segmentation of brain structures (cerebelum and each cerebral hemisphere without brain stem)
in MR images using CSM with IFTSC delineation.

Fourth Module
After brain segmentation, we will present a new method to correction MR intensities,
due to the inhomogeneity of the magnetic field, and segment brain tissues [2]. The resulting
brain voxels are finally classified into cerebral spinal fluid (CSF), gray matter (GM) and white
matter (WM), using IFT-based clustering and our knowledge about the order relation between
the intensities of these substances in different protocols (T1, T2 and PD). This knowledge
is exploited to assign class labels to the clusters. We will present the IFT-based clustering
approach and the algorithm for brain tissue segmentation with software demonstration.

Material
The only required equipment will be a digital projector, because we will take our notebook
with softwares and the pdf file of the slides.

Bibliography
[1] Y. Boykov and M.-P. Jolly. Interactive graph cuts for optimal boundary & region
segmentation of objects in N-D images. In International Conference on Computer
Vision, volume 1, pages 105112, 2001.
[2] F. A. M. Cappabianco, A. X. Falcao, Y. C. L. and J. K. Udupa MR-image segmentation
of brain tissues based on bias correction and optimum-path forest clustering.
Technical Report IC-10-07, 2010.
[3] A. X. Falcao, J. K. Udupa and F. K. Miyazawa. An ultra-fast user-steered image
segmentation paradigm: Live-wire-on-the-fly. IEEE Transactions on Medical Imaging,
19(1):5562, 2000.
[4] A. X. Falcao, J. Stolfi and R. A. Lotufo. The image foresting transform: Theory,
algorithms, and applications. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 26(1):1929, 2004.
[5] A. X. Falcao and F. P. G. Bergo. Interactive volume segmentation with differential
image foresting transforms. IEEE Trans. on Medical Imaging, 23(9):11001108, 2004.
[6] R. Minetto, J. P. Papa, T. V. Spina, A. X. Falcao, N. J. Leite and J. Stolfi. Fast and robust
object tracking using image foresting transform. In 16th International Conference on
Systems, Signals and Image Processing (IWSSIP), Chalkida, Greece, 2009. IEEE.
[8] P. A. V. Miranda, A. X. Falcao and J. K. Udupa. Cloud bank: a multiple clouds model
and its use in MR brain image segmentation. In 6th IEEE International Symposium
on Biomedical Imaging (ISBI), pages 506509, Boston, USA, 2009. IEEE.
[9] P. A. V. Miranda, A. X. Falcao and J. K. Udupa. Cloud models: Their construction and
employment in automatic MRI segmentation of the brain. Institute of Computing,
University of Campinas, Technical Report IC-10-08, Mar 2010.
[10] P. A. V. Miranda and A. X. Falcao. Links between image segmentation based on
optimum-path forest and minimum cut in graph. Journal of Mathematical Imaging
and Vision, 35(2):128142, October 2009.
[11] P. A. V. Miranda, A. X. Falcao and J. K. Udupa. Synergistic Arc-Weight Estimation
for Interactive Image Segmentation using Graphs. Computer Vision and Image
Understanding, 114(1):8599, Jan 2010.
[12] T. V. Spina, J. A. Montoya-Zegarra, A. X. Falcao and P. A. V. Miranda. Fast interactive
segmentation of natural images using the image foresting transform. In 16th
International Conference on Digital Signal Processing (DSP), pages 9981005, Santorini,
Greece, 2009. IEEE.

Alexandre Falc
ao, Thiago Spina, Paulo Miranda, F
abio Cappabianco
http://www.ic.unicamp.br/afalcao/
Dr. Alexandre X. Falc
ao is Associate Professor at the Institute of Computing,
University of Campinas (UNICAMP), SP, Brazil. He received a B.Sc. in Electrical Engineering
(1988) from the Federal University of Pernambuco (UFPE), PE, Brazil. He has worked in
image processing and analysis since 1991. In 1993, he received a M.Sc. in Electrical Engineering
from UNICAMP. During 1994-1996, he worked at the University of Pennsylvania, PA, USA, on
interactive image segmentation for his doctorate. He got his doctorate in Electrical Engineering
from UNICAMP in 1996. In 1997, he developed video quality assessing methods for TV Globo,
RJ, Brazil. He has been Professor at the Institute of Computing, UNICAMP, since 1998
and has published over 100 works on topics involving image processing and analysis, volume
visualization, content-based image retrieval, mathematical morphology, pattern recognition,
and medical imaging applications.
Thiago V. Spina received a B.Sc. in Computer Science (2009) from the University of
Campinas (UNICAMP), SP, Brazil. Since 2010 he has been working on natural images editing
for his Masters degree in Computer Science at the University of Campinas. His research involves
primarily segmentation and matting of images and videos using graph-based tools. His areas
of interest are image processing and analysis, computer vision, 3d reconstruction from videos,
and content based image/video retrieval.
Dr. Paulo A. V. Miranda received a B.Sc. in Computer Engineering (2003) and a M.Sc.
in Computer Science (2006) from the University of Campinas (UNICAMP), SP, Brazil. During
2008-2009, he worked at the University of Pennsylvania, PA, USA, on image segmentation
for his doctorate. He got his doctorate in Computer Science from the University of Campinas
(UNICAMP) in 2009. He is currently a post-doctoral researcher working in the project called,
BIA - Brain Image Analyzer, which is held in conjunction with professors of the Department
of Neurology, Unicamp, under the supervision of Prof. Falcao. His research involves image
segmentation and analysis, medical imaging applications, pattern recognition and content-based
image retrieval.
Dr. Fabio Cappabianco received a B.Sc. in Computer Engineering (2003) and a M.Sc.
in Computer Science (2006) from the University of Campinas (UNICAMP), SP, Brazil. During
2008-2009, he worked at the University of Pennsylvania, PA, USA, on image segmentation
for his doctorate. He got his doctorate in Computer Science from the University of Campinas
(UNICAMP) in 2010. He is currently lecturing at Foundation Hermnio Ometto. He is also a
collaborator in the project called, BIA - Brain Image Analyzer, which is held in conjunction with
professors of the Department of Neurology, Unicamp, under the supervision of Prof. Falcao. His
research involves medical image segmentation and analysis, pattern recognition and computer
architectures for image processing.