Eitan Sharon, CVPR04

Segmentation by Weighted Aggregation

Eitan Sharon
Division of Applied Mathematics,
Brown University
Eitan Sharon, Meirav Galun, Ronen Basri, Achi Brandt
Hierarchical Adaptive Texture Segmentation
or
Segmentation by Weighted Aggregation (SWA)
Dept. of CS and Applied Mathematics,
The Weizmann Institute of Science,
Rehovot, Israel
Eitan Sharon, CVPR04
Image Segmentation
Eitan Sharon, CVPR04
Local Uncertainty
Eitan Sharon, CVPR04
Global Certainty
Eitan Sharon, CVPR04
Local Uncertainty
Eitan Sharon, CVPR04
Global Certainty
Eitan Sharon, CVPR04
Hierarchy in SWA
Eitan Sharon, CVPR04
Segmentation by Weighted Aggregation
A multiscale algorithm:
Optimizes a global measure
Returns a full hierarchy of segments
Linear complexity
Combines multiscale measurements:
Texture
Boundary integrity
Eitan Sharon, CVPR04
The Pixel Graph
Couplings
Reflect intensity similarity
Low contrast
strong coupling
High contrast
weak coupling
{ }
ij
w
Eitan Sharon, CVPR04
Normalized-Cut Measure

=
S i
S i
u
i
0
1
Eitan Sharon, CVPR04
Normalized-Cut Measure
2
( ) ( )
ij i j
i j
E S w u u

=
S i
S i
u
i
0
1
Eitan Sharon, CVPR04
Normalized-Cut Measure
2
( ) ( )
ij i j
i j
E S w u u

=
S i
S i
u
i
0
1
( )
ij i j
N S w u u =

Eitan Sharon, CVPR04

Normalized-Cut Measure
2
( ) ( )
ij i j
i j
E S w u u

=
S i
S i
u
i
0
1
( )
ij i j
N S w u u =

( )
( )
( )
E S
S
N S
=
Minimize:
Eitan Sharon, CVPR04
Normalized-Cut Measure
High-energy cut
Minimize:
( )
( )
( )
E S
S
N S
=
Eitan Sharon, CVPR04
Normalized-Cut Measure
Low-energy cut
Minimize:
( )
( )
( )
E S
S
N S
=
Eitan Sharon, CVPR04
Matrix Formulation
,( )
ik
k k i
ij
ij
w i j
l
w i j

Define matrix by
Define matrix by
W 0
ij
w >
0
ii
w =
L
We minimize
( )
1
2
T
T
u Lu
u
u Wu
=
Eitan Sharon, CVPR04
Coarsening the Minimization Problem
i
u
j
u
Eitan Sharon, CVPR04
Coarsening Choosing a Coarse Grid
i
u
j
u
l
U
k
U
Representative subset
1 2
( , ,..., )
N
U U U U =
Eitan Sharon, CVPR04
Coarsening Interpolation Matrix
i
u
j
u
2
1
1
2
.
.
.
N
n
u
U
P
u
U
u
U
| |
| |
|
|
|
|
|
|
|
|
|
|
\ .
|
\ .

For a salient segment

(globally minimizing solutions):
P
( ) n N
, sparse interpolation matrix
k
U
l
U
Eitan Sharon, CVPR04
Coarsening Matrix Formulation
( )
( )
( )
1 1
2 2
T T
T
T T T
U P LP U
u Lu
u
u Wu U P WP U
=
Given such an appropriate interpolation matrix
P
Existence of from Algebraic Multigrid (AMG)
for solving the equivalent Eigen-Value problem:
P
Lu Du =
for minimal
0 >
solve
Eitan Sharon, CVPR04
Weighted Aggregation
ij
w
i
j
jl
p
aggregate
k
aggregate
l
ij k ik jl
i j
l
p p w W

kl
W
ik
p
Eitan Sharon, CVPR04
Importance of Soft Relations
Eitan Sharon, CVPR04
SWA
Linear in # of points
(a few dozen operations per point)
Detects the salient segments
Hierarchical structure
Eitan Sharon, CVPR04
Coarse-Scale Measurements
Average intensities of aggregates
Multiscale intensity-variances of aggregates
Multiscale shape-moments of aggregates
Boundary alignment between aggregates
Eitan Sharon, CVPR04
Coarse Measurements for Texture
Eitan Sharon, CVPR04
A Chicken and Egg Problem
Problem:
Coarse measurements
mix neighboring statistics
Solution:
Support of measurements
is determined as the
segmentation process
proceeds
Hey, I was
here first
Eitan Sharon, CVPR04
Adaptive vs. Rigid Measurements
Averaging
Our algorithm - SWA
Geometric
Original
Eitan Sharon, CVPR04
Our algorithm - SWA
Adaptive vs. Rigid Measurements
Interpolation
Geometric
Original
Eitan Sharon, CVPR04
Recursive Measurements: Intensity
i
ik
p
k
aggregate
k
j
i
q
intensity of pixel i
ik i
i
k
ik
i
p q
Q
p
=

average intensity
of aggregate
Eitan Sharon, CVPR04
Use Averages to Modify the Graph
Eitan Sharon, CVPR04
Hierarchy in SWA
Average intensity
Texture
Shape
Eitan Sharon, CVPR04
Texture Examples
Eitan Sharon, CVPR04
Isotropic Texture in SWA
( ) ( )
2
2
k k
k
V I I =
Intensity Variance
Isotropic Texture of aggregate
average of variances in all scales
Eitan Sharon, CVPR04
Oriented Texture in SWA
Shape Moments
Oriented Texture of aggregate
orientation, width and length in all scales
2 2
, , , , x y xy x y
center of mass
width
length
orientation
with Meirav Galun
Eitan Sharon, CVPR04
Implementation
Our SWA algorithm (CVPR00 + CVPR01 + orientations Texture)
run-time: << 1 seconds.
images on a Pentium III 1000MHz PC: 200 200
400 400
run-time: 2-3 seconds.
Eitan Sharon, CVPR04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR04
Our Algorithm
(SWA)
Our previous
Eitan Sharon, CVPR04
Our Algorithm (SWA)
Our previous
Eitan Sharon, CVPR04
Our Algorithm (SWA)
Our previous
Eitan Sharon, CVPR04
Our Algorithm (SWA)
Our previous