Sei sulla pagina 1di 18

Motion estimation for video compression

Blockmatching Search strategies for block matching Block comparison speedups Hierarchical blockmatching Sub-pixel accuracy

  Hierarchical blockmatching    Sub-pixel accuracy Bernd Girod: EE398A Image and Video Compression Motion

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 1

Block-matching algorithm search range in previous frame S k − 1      

Block-matching algorithm

search range in previous frame

S k 1

Subdivide current frame into blocks. Find one displacement vector for each block. Within a search range, find a “best match” that minimizes an error measure. Intelligent search strategies can reduce computation.

range, find a “best match” that minimizes an error measure. Intelligent search strategies can reduce computation.
range, find a “best match” that minimizes an error measure. Intelligent search strategies can reduce computation.
range, find a “best match” that minimizes an error measure. Intelligent search strategies can reduce computation.
Intelligent search strategies can reduce computation. block of current frame S k Bernd Girod: EE398A Image
Intelligent search strategies can reduce computation. block of current frame S k Bernd Girod: EE398A Image
Intelligent search strategies can reduce computation. block of current frame S k Bernd Girod: EE398A Image

block of current

frame

S k

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 2

Block-matching algorithm Previous Frame
Block-matching algorithm
Previous Frame
Block-matching algorithm Previous Frame Current Frame Measurement window is 
 compared with a shifted block 

Current Frame
Current Frame
Block-matching algorithm Previous Frame Current Frame Measurement window is 
 compared with a shifted block 


Measurement window iscompared with a shifted blockof pixels in the other image, to determine the best match

Block of pixels is selected as a measurement window

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 3

Block-matching algorithm

Previous Frame
Previous Frame
Current Frame
Current Frame

process repeated for another block.

Frame Current Frame process repeated for another block. Bernd Girod: EE398A Image and Video Compression Motion

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 4

Blockmatching: Matching Criterion   Sum of Squared Differences to determine similarity Sum all values in
Blockmatching: Matching Criterion
  Sum of Squared Differences to determine similarity
Sum all values
in measurement
window
Current
Previous
image
image

Horizontal

shift

window Current Previous image image Horizontal shift Vertical shift    Alternative matching criteria: SAD
window Current Previous image image Horizontal shift Vertical shift    Alternative matching criteria: SAD

Vertical

shift

Alternative matching criteria: SAD (Sum of Absolute Differences), cross correlation, How about sub-pixel shifts?

), cross correlation,    How about sub-pixel shifts? Bernd Girod: EE398A Image and Video Compression

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 5

SSD Values Resulting from Blockmatching

SSD Values Resulting from Blockmatching Estimated displacement Integer-pixel accuracy Vertical shift d y 6 -8 4
Estimated displacement Integer-pixel accuracy Vertical shift d y 6 -8 4 -3 2 0 2
Estimated displacement
Integer-pixel accuracy
Vertical shift
d y
6
-8
4
-3
2
0
2
-2
-4
7 -6
Horizontal shift
d
x

SSD

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 6

Motion-compensated prediction: example

Previous frame

Current frame

prediction: example Previous frame Current frame Prediction with displacement vectors Motion-compensated
prediction: example Previous frame Current frame Prediction with displacement vectors Motion-compensated

Prediction with

displacement

vectors

Motion-compensated Prediction error

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 7

SSD Sub-pixel Accurate Minimum
SSD
Sub-pixel
Accurate
Minimum

Horizontal shift d x

Interpolation of the SSD Minimum

Fit parabola

Fit parabola

through

through

>3 points

3 points

approximately

exactly


 through >3 points 3 points approximately exactly Bernd Girod: EE398A Image and Video Compression Motion

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 8

2-d Interpolation of SSD Minimum

Paraboloid Perfect fit through 6 points Approximate fit through >6 points

SSD Horizontal shift d x
SSD
Horizontal shift d x

 >6 points       SSD Horizontal shift d x Bernd Girod: EE398A Image

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 9

Blockmatching: search strategies I

Full search

All possible displacements within the search range are compared. Computationally expensive Highly regular, parallelizable

• •

• •

• •

• •

• •

• •

• •

• •

• •

• •

• •

• •

• •

• • • • • • • • • Bernd Girod: EE398A Image and Video Compression
• • • • • • • • • Bernd Girod: EE398A Image and Video Compression
• • • • • • • • • Bernd Girod: EE398A Image and Video Compression

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 10

Blockmatching: search strategies II

2D logarithmic search [Jain + Jain, 1981]

Iterative comparison of error measure values at 5 neighboring points Logarithmic refinement of the search pattern if

best match is in the center of the 5- point pattern center of search pattern touches the border of the search range

of search pattern touches the border of the search range Bernd Girod: EE398A Image and Video
of search pattern touches the border of the search range Bernd Girod: EE398A Image and Video

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 11

Blockmatching: search strategies III Diamond search [Li, Zeng, Liou, 1994] [Zhu, Ma, 1997] Start with

Blockmatching: search strategies III

Diamond search [Li, Zeng, Liou, 1994] [Zhu, Ma, 1997]

III Diamond search [Li, Zeng, Liou, 1994] [Zhu, Ma, 1997] Start with 
 large diamond 


Start with large diamondpattern at

(0,0)

If best matchlies in the center of large diamond, proceed with small diamond

If best match does not lie in the center of large diamond, center large diamond pattern at new best match

center large diamond pattern at new 
 best match Bernd Girod: EE398A Image and Video Compression

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 12

Blockmatching: search strategies IV

Most search strategies can be further accelerated by Predictive motion search

Use median of motion vectors in causal neighborhood as starting point for search. Additionally test zero-vector as a starting point

Early termination

Interrupt summation to calculate SSD or SAD, if value grows too quickly (relative to previous best match) Stop search, if match is “good enough” (SSD, SAD < threshold)

if match is “good enough” (SSD, SAD < threshold) Bernd Girod: EE398A Image and Video Compression

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 13

Block comparison speed-ups Triangle and Cauchy-Schwarz inequalities for SAD and SSE   ∑ ≥ =
Block comparison speed-ups
Triangle and Cauchy-Schwarz inequalities for SAD and SSE
 
=
S
S k − S k −1
∑ S k − S k −1
k
∑ S k −1
block
block
block
block
2
1
1
(
)
2 ≥
S k
 2 =
S k
S k −1
∑ S k − S k −1
∑ S k −1
N 
N 
block
block
block
block
number of terms in sum
Strategy:
 

Compute partial sums for blocks in current and previous frame Compare blocks based on partial sums Omit full block comparison, if partial sums indicate worse error measure than previous best result

Performance: > 20x speed-up of full search block matching reported by

employing

[Lin + Tai, IEEE Tr. Commun., May 97]

reported by employing [Lin + Tai, IEEE Tr. Commun., May 97]    Sum over 16x16

Sum over 16x16 block Row wise block projection Column wise block projection

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 14

Hierarchical blockmatching

Integer-pixel accuracy

Extension to sub-pixel accuracy

Integer-pixel accuracy Extension to sub-pixel accuracy Bernd Girod: EE398A Image and Video Compression Motion
Integer-pixel accuracy Extension to sub-pixel accuracy Bernd Girod: EE398A Image and Video Compression Motion

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 15

      • • • • • • • • • • • •
 
 
 
• •
• •
• •
• •
• •
• •
• •
• •
• •
• •
• •
• •
• •
• •
• •
• •
• •

Sub-pixel accuracy

Interpolate pixel raster of the reference image to desired sub-pixel accuracy (typically by bi-linear interpolation) Straightforward extension of displacement vector search to fractional accuracy Example: half-pixel accurate displacements

accuracy Example: half-pixel accurate displacements Bernd Girod: EE398A Image and Video Compression Motion
accuracy Example: half-pixel accurate displacements Bernd Girod: EE398A Image and Video Compression Motion
accuracy Example: half-pixel accurate displacements Bernd Girod: EE398A Image and Video Compression Motion

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 16

Bi-linear Interpolation brightness
Bi-linear Interpolation
brightness
Bi-linear Interpolation brightness Interpolated Pixel Value Bernd Girod: EE398A Image and Video Compression Motion

Interpolated Pixel Value

Bi-linear Interpolation brightness Interpolated Pixel Value Bernd Girod: EE398A Image and Video Compression Motion

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 17

Reading

J. R. Jain, A. K. Jain, “Displacement Measurement and Its Application in Interframe Image Coding,” IEEE Trans. Communications, vol. COM-29, no. 12, pp. 1799-1808, Dec.

1981.

Y.-C. Lin, S.-C. Tai, “Fast Full-Search Block-Matching Algorithm for Motion-Compensated Video Compression,” IEEE Trans. Communications, vol. 45, no. 5, pp. 527-531, May 1997.

Communications, vol. 45, no. 5, pp. 527-531, May 1997. Bernd Girod: EE398A Image and Video Compression

Bernd Girod: EE398A Image and Video Compression

Motion estimation no. 18