Tutorial

Graph Based Image Segmentation

Jianbo Shi, David Martin, Charless Fowlkes, Eitan Sharon

 Topics

• Computing segmentation with graph cuts

• Segmentation benchmark, evaluation criteria
• Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation
 i
Wij
j

Wij

G = {V, E}

V: graph nodes Image = { pixels }

E: edges connection nodes Pixel similarity
 Topics

• Computing segmentation with graph cuts

• Segmentation benchmark, evaluation criteria
• Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation
 Topics

• Computing segmentation with graph cuts

• Segmentation benchmark, evaluation criteria
• Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation
 Texture Brightnes Colo
Original Image s r
a*
L*
b*

Boundary Processing Region

Processing

Textons

D Proximity C A
E
B

χ2 χ2

Wij
C A

B
 Topics

• Computing segmentation with graph cuts

• Segmentation benchmark, evaluation criteria
• Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation
Part I: Graph and Images

Jianbo Shi
Graph Based Image Segmentation

i
Wij
j

Wij

G = {V, E}

V: graph nodes Image = { pixels }

E: edges connection nodes Pixel similarity

Segmentation = Graph partition

 Right partition cost function?

Efficient optimization algorithm?

Graph Terminology
adjacency matrix,
degree,
volume,
graph cuts
 Graph Terminology
i
Sij
j
Similarity matrix S = [ Sij ]
is generalized adjacency matrix

0.9
20 20
0.8

40 0.7
10
0.6
60
0.5
0
80
0.4

100 0.3
!10

0.2

120
!20 0.1

!10 0 10 20 30 40
140 0
20 40 60 80 100 120 140
 Graph Terminology
! i
Degree of node: di = Sij
j

0.9
20 20
0.8

40 0.7
10
0.6
60
0.5
0
80
0.4

100 0.3
!10

0.2

120
!20 0.1

!10 0 10 20 30 40
140 0
20 40 60 80 100 120 140
 Graph Terminology
Volume of set:
!
vol(A) = di , A ⊆ V
i∈A A

0.9
20 20
0.8

40 0.7
10
0.6
60
0.5
0
80
0.4

100 0.3
!10

0.2

120
!20 0.1

!10 0 10 20 30 40
140 0
20 40 60 80 100 120 140
 Cuts in a graph
!
cut(A, Ā) = Si,j
i∈A,j∈Ā

0.9
20 20
0.8

40 0.7
10
0.6
60
0.5
0
80
0.4

100 0.3
!10

0.2

120
!20 0.1

!10 0 10 20 30 40
140 0
20 40 60 80 100 120 140
 Graph Terminology
!
Similarity matrix S = [ Sij ] Degree of node: di = Sij
i
Sij j
j i

Volume of set: Graph Cuts

A
 Useful Graph Algorithms

• Minimal Spanning Tree

• Shortest path
• s-t Max. graph flow, Min. cut
 Minimal/Maximal Spanning Tree

Tree is a graph G without cycle

Graph Maximal Minimal

 Kruskal’s algorithm
• sort the edges of G in increasing order by length
• for each edge e in sorted order
if the endpoints of e are disconnected in S
add e to S

Randonmalized version can compute Typical cuts

Leakage problem in MST

Leakage
 Graph Cut and Flow

Source
Sink

1) Given a source (s) and a sink node (t)

2) Define Capacity on each edge, C_ij = W_ij
3) Find the maximum flow from s->t, satisfying the capacity constraints

Min. Cut = Max. Flow

Problem with min cuts

Min. cuts favors isolated clusters

 Normalize cuts in a graph

• (edge) Ncut = balanced cut

1 1
N cut(A, B) = cut(A, B)( + )
vol(A) vol(B)

NP-Hard!
 Representation
Partition matrix: segments

X = [X1 , ..., XK ]
pixels

Pair-wise similarity matrix W

Laplacian matrix D-W
!
Degree matrix D: D(i, i) = Wi,j
j
Graph weight matrix W
(b)
(a) W
i1
n=nr * nc

nr
i2

nc

n=nr * nc
W(i1,j) (c) W(i2,j)
(d)
 Laplacian matrix D-W
Let x = X(1,:) be the indicator of group 1

asso(A, A) = xT W x

xT W x
1

0.9
20
0.8

40 0.7

0.6
60
0.5

80
0.4

100 0.3

0.2

120
0.1

140 0
20 40 60 80 100 120 140
 Laplacian matrix D-W
T T
x Dx x Wx
Cut(A,V-A) = vol(A) - asso(A,A)
1

0.9
20
0.8

40 0.7

0.6
60
0.5

80
0.4

100 0.3

0.2

120
0.1

140 0
20 40 60 80 100 120 140

Cut(A, V − A) = xT (D − W )x
 1 !
K
cut(Vl , V − Vl )
N cut(X) =
K vol(Vl )
l=1

0.9
20
0.8

40 0.7

0.6
60
0.5

80
0.4

100 0.3

0.2

120
0.1

140 0
20 40 60 80 100 120 140

1 !K
XlT (D − W )Xl
= T DX
K X l l
l=1

X ∈ {0, 1}N ×K , X1K = 1N

 Step I: Find Continuous Global Optima

T − 12
Scaled partition matrix. Z = X(X DX)
1
 
  √ 0 0
1 0 0 vol(A)
√ 1
 
0 0
1 0 0
 
   vol(A) 
  1
0 0
 
X=
 0 1 0 
 Z=

√
vol(B)


 0 1 0  
 0 √ 1 0 

vol(B)
0 0 1
 
1
0 0 √
vol(C)
 Step I: Find Continuous Global Optima

1 !K
XlT (D − W )Xl
Ncut = T DX
K X l l
l=1

becomes
1
N cut(Z) = tr(Z T W Z) Z T DZ = IK
K
Eigensolutions
y2i
i
(D − W )z = λDz ∗ ∗
A

y2i
Z =∗
[z1∗ , z2∗ , ..., zk∗ ] i
A
Interpretation as a Dynamical System
Interpretation as a Dynamical System
 Step I: Find Continuous Global Optima

Partition Scaled Partition Eigenvector solution

1
 
0 0
   1

1 0 0 √
vol(A) √
vol(A)
0 0
√ 1
 
 1 0 0   0 0  
√ 1 0 0

vol(A)  
vol(A)
 
1
    ' (
0 0  × cos(θ) -sin(θ)
 
X= 0 1 0  Z= √
vol(B)
 
Z =
∗ 0 √ 1
vol(B)
0

.
sin(θ) cos(θ)
   
√ 1
 
 0 1 0  
 0 0 
  0 √ 1 0 
 vol(B)   vol(B)

1
0 0 1 0 0
 
1
√
vol(C) 0 0 √
vol(C)

(D − W )Z ∗ = λDZ ∗

T
If Z* is an optimal, so is {ZR : R R = IK }
 Step II: Discretize Continuous Optima
 
1 0
1 0
Target
 
 
X= 0 1 
partition
 
Z2  0
0
1
1


Z1
Rotation R
 
1 -1.4
1 -1.3
Eigenvector Z ∗ = 
 

 1 0.8 
solution 
 1 0.9


1 0.7

Rotation R can be found exactly in 2-way partition

Brightness Image
Segmentation
brightness image
segmentation
 Part II:
Segmentation Measurement,
Benchmark