UpdateOnFeatures StefanHaller

AN UPDATE ON FEATURES
SURF, BRISK, ORB and FREAK
Stefan Haller ↦ stefan.haller@tu-dresden.de
Dresden, 12th January 2015

Outline
1. Introduction to Features
2. SURF – Speeded Up Robust Features
3. BRISK – Binary Robust Invariant Scalable Keypoints
4. ORB – Oriented FAST and Rotated BRIEF
5. FREAK – Fast Retina Keypoints
6. Comparison
7. References
8. Discussion
TU Dresden, 12th January 2015 An Update on Features Folie 2 von 51

Introduction to Features

Image Features
achievement:
• detect image features: unique interest points
• describe image feature: sample region of image patch
around point
applications include:
• object recognition and tracking
• image matching and stitching
• robotic mapping and 3D modeling
important:
• repeatability and reliability
• good performance (description, matching)
• speed
1st Step: Feature Detection
By Retardo
(http://en.wikipedia.org/
wiki/File:Corner.png)
[Public domain], via
Wikimedia Commons

2nd Step: Feature Description
Bay, H., Tuytelaars, T., & Van

Gool, L. (2006). Surf: Speeded
up robust features. In
Computer Vision–ECCV
2006 (pp. 404-417). Springer
Berlin Heidelberg.
Lowe, D. G. (2004).
Distinctive image features
from scale-invariant
keypoints. International
journal of computer vision,
60(2), 91-110.

History
Traditional (slower, accurate):

1999 Scale Invariant Feature Transform (Lowe)
2006 Speeded Up Robust Features (Bay, Tuytelaars, Van Gool)

History
Traditional (slower, accurate):

1999 Scale Invariant Feature Transform (Lowe)
2006 Speeded Up Robust Features (Bay, Tuytelaars, Van Gool)
Binary (faster, real time, smartphone, performance):

2010 Binary Robust Independent Elementary Features
(Michael Calonder, et al.)
2011 Oriented FAST and Rotated BRIEF (Ethan Rublee et al.)
2011 Binary Robust Invariant Scalable Keypoints
(Leutenegger, Chli, Siegwart)
2012 Fast Retina Keypoint (Alahi, Ortiz, Vandergheynst)

SURF – Speeded Up
Robust Features

Properties
• scale and rotation invariant (Upright SURF only scale

invariant)
• faster computation and faster matching
• achieved by integral images and smaller feature vector size
• ideas of SIFT, but much more simplified

Feature Detection: Fast-Hessian
• based on Hessian matrix (Laplacian of Gaussian, LoG)
• reminder: LoG approximated by Difference of Gaussian,
DoG
• SURF uses “Fast-Hessian Detector”: approximation with
box filters

DoG
box filters

DoG
box filters
box filters → very fast for arbitrary size (integral image method)
Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. In Computer
Vision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.

Feature Detection: Scale invariance
• use of scale space pyramid
• box filters allow to blur base image of current octave
• parallel creation of scales for octave possible!
Lowe, D. G. (2004).
Distinctive image features
from scale-invariant
keypoints. International
journal of computer vision,
60(2), 91-110.

Feature Description: Orientation
Assignment
• (step is skipped for USURF)
• consider circular region around image patch (size
dependes on scale)
• calculate Haar wavelet responses in 𝑥 and 𝑦 direction
(integral images → 6 ops for any sampling point)
1.5
1.0
0.5 “Haar wavelet”. Licensed under CC BY-SA 3.0 via

0.0
Wikimedia Commons -
-0.5
http://commons.wikimedia.org/wiki/
File:Haar_wavelet.svg
-1.0
-1.5
0 1
• representation in vector space

• sliding window → sum up responses → choose largest
vector
Feature Description: Dominant Direction
vertical
response
horizontal
response

vertical
response
horizontal
response

vertical
response
horizontal
response

vertical
response
horizontal
response

vertical
response
horizontal
response

vertical
response
horizontal
response

vertical
response
horizontal
response

Feature Description Vector
• square region around IP (rotated)

• split into 4 × 4 subregion
• for each subregion: 5 × 5 regularly spaced sample points
• compute Haar wavelet response horiz. and vert.
• for each: sum of d𝑥, d𝑦, |d𝑥|, |d𝑦|
→ 4 × 4 region, respectively 4 dim. vector → 64 dim. vector

Feature Description Vector
Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. In Computer
Vision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.

BRISK – Binary Robust
Invariant Scalable
Keypoints

Properties
• dramatically lower computational complexity

• assembly of bit-string vector (no gradients, etc!)
• general property of binary descriptors: use of Hamming
distance instead of Euclidean distance
• Hamming distance ≙ XOR both vectors and count “1”s
→ boosted by SSE/AVX
• faster than SIFT/SURF, comparable performance
(binary descriptors) especially suited for:

• real time requirements
• low power devices

Scale-Space Keypoint Detection
• extension of AGAST which is itself extension FAST

• invariant of scale
• estimates keypoint scale in continuous scale-space

Keypoint Description
• composed of binary string, simple brightness comparison
tests
• neighborhood sampling: deterministic, equally spaced,
concentric circles
Leutenegger, S., Chli, M., & Siegwart,

R. Y. (2011, November). BRISK:
Binary robust invariant scalable
keypoints. In Computer Vision
(ICCV), 2011 IEEE International
Conference on (pp. 2548-2555). IEEE.
Long and Short Pairs
𝒜 = {(𝐩𝑖 , 𝐩𝑗 ) ∈ ℝ2 × ℝ2 ∣ 𝑖, 𝑗 ∈ ℕ, 𝑗 < 𝑖 < 𝑁 }
𝒮 = {(𝐩𝑖 , 𝐩𝑗 ) ∈ 𝒜 ∣ ‖𝐩𝑗 − 𝐩𝑖 ‖ < 𝛿𝑚𝑎𝑥 } ⊆ 𝒜

ℒ = {(𝐩𝑖 , 𝐩𝑗 ) ∈ 𝒜 ∣ ‖𝐩𝑗 − 𝐩𝑖 ‖ > 𝛿𝑚𝑖𝑛 } ⊆ 𝒜
Leutenegger, S., Chli, M., & Siegwart,

R. Y. (2011, November). BRISK:
Binary robust invariant scalable
keypoints. In Computer Vision
(ICCV), 2011 IEEE International
Orientation Computation
• long distance pairs used
• calculate local gradient between pairs
• sum up all local gradients
𝐼(𝐩𝑗 , 𝜎𝑗 ) − 𝐼(𝐩𝑖 , 𝜎𝑖 )
𝐠(𝐩𝑖 , 𝐩𝑗 ) = (𝐩𝑗 − 𝐩𝑖 ) ⋅
‖𝐩𝑗 − 𝐩𝑖 ‖2
𝑔 1
𝐠 = ( 𝑥 ) = ⋅ ∑ 𝐠(𝑝𝑖 , 𝑝𝑗 )
𝑔𝑦 𝐿 (𝐩 ,𝐩 )∈ℒ
𝑖 𝑗
𝜃 = arctan2(𝑔𝑦 , 𝑔𝑥 )
Building the Descriptor
• binary descriptor: comparison of points with binary

outcome
• short distance pairs used
1 if 𝐼(𝐩𝛼 𝛼
𝑗 , 𝜎𝑗 ) > 𝐼(𝐩𝑖 , 𝜎𝑖 )
𝑏={
0 otherwise
∀(𝐩𝛼 𝛼
𝑖 , 𝐩𝑗 ) ∈ 𝒮
• Hamming distance for matching

• XOR two vectors and count resulting “1”s

ORB – Oriented FAST
and Rotated BRIEF

Properties
• fast, computational efficient

• real time and low power device
• (all other properties of binary descriptors)

Detector (oriented FAST)
• modified FAST: oriented FAST

• measures intensity between center pixel and those in
circular ring around center
shortcomings of FAST approach:

1. no quality measure (“cornerness”)
solution: use Harris cornerness measure
2. not scale invariant
solution: use scale pyramid (like SIFT)
3. not rotation invariant
solution: calculate intensity centroid

Intensity Centroid (Rotation Invariance)
assumption: corner’s intensity is offset from center
∀ 𝑝, 𝑞⏟
∈ {0, 𝑝 𝑦𝑞
⏟⏟ ⏟1}⏟∶ 𝑚𝑝𝑞 = ∑ 𝑥
⏟ 𝐼(𝑥,
⏟ 𝑦)
binary selector 𝑥,𝑦
⏟ weighted image
for x and y circular by function
direction window coordinate
𝑚10 𝑚01
𝐶=( , )
𝑚00 𝑚00
dominant direction ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗

𝑂𝐶: 𝜃 = atan2(𝑚01 , 𝑚10 )

Intensity Weighted Center of Mass

Descriptor (modified BRIEF)
• binary descriptor, no sampling pattern
1 if 𝑝(𝑥) ≥ 𝑝(𝑦)
• 𝜏 (𝑝; 𝑥, 𝑦) = {
0 otherwise
• feature vector 𝑓𝑛 (𝑝) ≔ ∑ 2𝑖−1 𝜏 (𝑝; 𝑥𝑖 , 𝑦𝑖 )

1≤𝑖≤𝑛
• vector length 𝑛 = 256 bit

Learning of Pairs
• 256 bit, but thousands of possible pairs

• Which pairs provide best information for given patch?
• We want to maximize variance and thus minimize
correlation.
• Paper suggests learning algorithm

• Input: reference image series / Output: static list of pairs
⌛ Not enough time to explain in detail

→ Discussion afterwards?

FREAK – Fast Retina
Keypoints

Properties and Motivation
• binary descriptor, no feature detector specified

• inspired by human visual system, precisely retina
• at same time enforcing low computational complexity
• tries to be faster and more robust than SIFT, SURF or
BRISK
• (all other properties of binary descriptors)

Biological Background
• human retina extracts details from images using DoGs
• differences are encoded into action potentials
• several photo-receptors influence ganglion cell
Alahi, A., Ortiz, R., &

Vandergheynst, P. (2012,
June). Freak: Fast retina
keypoint. In Computer
Vision and Pattern
Recognition (CVPR),
2012 IEEE Conference
on (pp. 510-517). Ieee.

Biological Background
• human retina extracts details from images using DoGs
• differences are encoded into action potentials
• several photo-receptors influence ganglion cell
• size and dendritic field increases with radial distance
from fovea
• spatial distribution of ganglion cells reduces exponentially
Alahi, A., Ortiz, R., &

Vandergheynst, P. (2012,
June). Freak: Fast retina
keypoint. In Computer
Vision and Pattern
Recognition (CVPR), 2012
IEEE Conference on (pp.
510-517). Ieee.
FREAK’s Sampling Pattern
• sampling pattern similar to retinal ganglion cells
• receptive fields are overlapping → redundancy
• increases performance, better discriminative power
Alahi, A., Ortiz, R., & Vandergheynst,

P. (2012, June). Freak: Fast retina
keypoint. In Computer Vision and
Pattern Recognition (CVPR), 2012 IEEE
Conference on (pp. 510-517). Ieee.
FREAK’s methods
⌛ skipped:
orientation: based on BRISK
sampling pairs: learning

algorithm based on ORB
the results are quite interesting:

coarse-to-fine ordering is
automatically preferred!
Alahi, A., Ortiz, R., & Vandergheynst, P. (2012,
June). Freak: Fast retina keypoint. In Computer
Vision and Pattern Recognition (CVPR), 2012 IEEE
Conference on (pp. 510-517). Ieee.

Saccadic Search
• eyes perform discontinuous individual movements
(“saccades”)
• fovea captures high-resolution → high density

• critical role during recognition and matching
• perifoveal area captures less detailed information

• used to compile first estimates
“Szakkad” by Original file: SpooSpa. Derivative:

Simon Viktória - Derivative work from File:Face
of SpooSpa.jpg. Licensed under CC BY-SA 2.0 via
Wikimedia Commons -
http://commons.wikimedia.org/wiki/
File:Szakkad.jpg
Saccadic Search
• FREAK performs several steps
• at first parse first 16 byte of descriptor → corresponds to

outer coarse information
• ≥ 90% of candidates discarded with first 16 bytes
• use SIMD: comparing 16 byte as fast as 1 byte

Comparison

Summary
real valued detector descriptor

SIFT LoG pyramid gradient values
SURF box filter pyramid Haar wavelets
binary detector sample pattern orientation sampling pairs

BRISK AGAST concentric circles gradient short pairs
ORB oFAST none moments learned
FREAK N/A overlapping con- gradient (pre- learned
centric circles selected pairs)

Performance Evaluation
• source: “Evaluation of Local Detectors and Descriptors for

Fast Feature Matching”
• comparison of matching speed and precision/accuracy

• only publicly available implementations can be tested
(→ OpenCV)

Implementation Details
• matching: nearest neighborhood search (NN)

1. kd-trees (geometric, partition of dimensions)
→ for real-valued descriptors
2. hashing (project similar data into same bucket, Hamming space)
→ for binary descriptors
• ϵ-ANN: approximated NN
• image data set: standard graffiti image sequences
• 24× 3.47 GHz, 12 MB cache, Intel Xeon X5690, 99 GB
RAM, x64 Ubuntu 10.04
• SSE 4.2 enabled (fast POPCNT)
• algorithms used with default parameters (except FAST)

Miksik, O., & Mikolajczyk, K. (2012, November). Evaluation of local detectors and descriptors for
fast feature matching. In Pattern Recognition (ICPR), 2012 21st International Conference on (pp.
2681-2684). IEEE.

2681-2684). IEEE.

2681-2684). IEEE.

Performance Measures
#true positive
recall =
#true positive + #false negative
#true positive
precision =
#true positive + #false positive

2681-2684). IEEE.

My Conclusion
• SIFT: slow + good quality
• BRISK: fast + good quality
• FREAK ≈ BRISK
• ORB: faster + littlebit less quality

References

References (Primary)
📖 Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. In
Computer Vision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.
📖 Rublee, E., Rabaud, V., Konolige, K., & Bradski, G. (2011, November). ORB: an efficient
alternative to SIFT or SURF. In Computer Vision (ICCV), 2011 IEEE International
📖 Leutenegger, S., Chli, M., & Siegwart, R. Y. (2011, November). BRISK: Binary robust
invariant scalable keypoints. In Computer Vision (ICCV), 2011 IEEE International
📖 Alahi, A., Ortiz, R., & Vandergheynst, P. (2012, June). Freak: Fast retina keypoint. In
Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (pp. 510-517).
Ieee.
📖 Miksik, O., & Mikolajczyk, K. (2012, November). Evaluation of local detectors and
descriptors for fast feature matching. In Pattern Recognition (ICPR), 2012 21st
International Conference on (pp. 2681-2684). IEEE.

References (Secondary)
📖 Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International

journal of computer vision, 60(2), 91-110.
📖 Rosten, E., & Drummond, T. (2006). Machine learning for high-speed corner detection. In
Computer Vision–ECCV 2006 (pp. 430-443). Springer Berlin Heidelberg.
📖 Juan, L., & Gwun, O. (2009). A comparison of sift, pca-sift and surf. International Journal of
Image Processing (IJIP), 3(4), 143-152.
📖 Calonder, M., Lepetit, V., Strecha, C., & Fua, P. (2010). Brief: Binary robust independent
elementary features. In Computer Vision–ECCV 2010 (pp. 778-792). Springer Berlin
Heidelberg.
📖 Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evaluation of binary keypoint descriptors.
In Image Processing (ICIP), 2013 20th IEEE International Conference (pp. 3652-3656). doi:
10.1109/ICIP.2013.6738753
📖 Figat, J., Kornuta, T., & Kasprzak, W. (2014). Performance Evaluation of Binary Descriptors
of Local Features. In Computer Vision and Graphics (pp. 187-194). Springer International
Publishing.

Discussion

Discussion
• real valued vs. binary descriptor

• simplicity vs. complexity
• learning of optimal pairs for ORB/FREAK
• Is evaluation of paper exhaustive?
• real time?
• object tracking?

Learning Algorithm (ORB)



Learning Algorithm (FREAK)

Two Supplementary Evaluation Papers
Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evalua- Figat, J., Kornuta, T., & Kasprzak, W. (2014). Performance
tion of binary keypoint descriptors. In Image Proces- Evaluation of Binary Descriptors of Local Features. In
sing (ICIP), 2013 20th IEEE International Conference (pp. Computer Vision and Graphics (pp. 187-194). Springer In-
3652-3656). doi: 10.1109/ICIP.2013.6738753 ternational Publishing.
compares BRIEF, ORB, BRISK and compares BRIEF, ORB, BRISK and
FREAK FREAK; each with all kinds of detectors
data sets: Oxford and Stanford Mobile data set: Oxford
Visual Search (the last w/o ground conclusion: recommends ORB detector
truth values) with BRISK descr. or FREAK descr.
conlusion: paper recommends BRISK (depending on distortion type)
BRISK needs significant more FREAK for viewpoint changes much
computational effort compared to ORB better than BRISK/ORB
FREAK faster than BRISK, has less SIFT has state-of-the-art performance
memory load, slightly less performance
SIFT has state-of-the-art performance

Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evaluation of binary keypoint descriptors. In Image
Processing (ICIP), 2013 20th IEEE International Conference (pp. 3652-3656). doi:
10.1109/ICIP.2013.6738753

Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evaluation of binary keypoint descriptors. In Image
Processing (ICIP), 2013 20th IEEE International Conference (pp. 3652-3656). doi:
10.1109/ICIP.2013.6738753

UpdateOnFeatures StefanHaller

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

UpdateOnFeatures StefanHaller

Caricato da

Copyright:

Formati disponibili

AN UPDATE ON FEATURES

SURF, BRISK, ORB and FREAK

Stefan Haller ↦ stefan.haller@tu-dresden.de

Dresden, 12th January 2015

2. SURF – Speeded Up Robust Features

3. BRISK – Binary Robust Invariant Scalable Keypoints

4. ORB – Oriented FAST and Rotated BRIEF

5. FREAK – Fast Retina Keypoints

TU Dresden, 12th January 2015 An Update on Features Folie 2 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 3 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 5 von 51

Bay, H., Tuytelaars, T., & Van

TU Dresden, 12th January 2015 An Update on Features Folie 6 von 51

Traditional (slower, accurate):

TU Dresden, 12th January 2015 An Update on Features Folie 7 von 51

Traditional (slower, accurate):

Binary (faster, real time, smartphone, performance):

TU Dresden, 12th January 2015 An Update on Features Folie 7 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 8 von 51

• scale and rotation invariant (Upright SURF only scale

• ideas of SIFT, but much more simplified

TU Dresden, 12th January 2015 An Update on Features Folie 9 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 10 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 10 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 10 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 11 von 51

0.5 “Haar wavelet”. Licensed under CC BY-SA 3.0 via

• representation in vector space

TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51

• square region around IP (rotated)

• for each: sum of d𝑥, d𝑦, |d𝑥|, |d𝑦|

→ 4 × 4 region, respectively 4 dim. vector → 64 dim. vector

TU Dresden, 12th January 2015 An Update on Features Folie 14 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 15 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 16 von 51

• dramatically lower computational complexity

(binary descriptors) especially suited for:

TU Dresden, 12th January 2015 An Update on Features Folie 17 von 51

• extension of AGAST which is itself extension FAST

TU Dresden, 12th January 2015 An Update on Features Folie 18 von 51

Leutenegger, S., Chli, M., & Siegwart,

𝒮 = {(𝐩𝑖 , 𝐩𝑗 ) ∈ 𝒜 ∣ ‖𝐩𝑗 − 𝐩𝑖 ‖ < 𝛿𝑚𝑎𝑥 } ⊆ 𝒜

Leutenegger, S., Chli, M., & Siegwart,

• binary descriptor: comparison of points with binary

• Hamming distance for matching

TU Dresden, 12th January 2015 An Update on Features Folie 22 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 23 von 51

• fast, computational efficient

• (all other properties of binary descriptors)

TU Dresden, 12th January 2015 An Update on Features Folie 24 von 51

• modified FAST: oriented FAST

shortcomings of FAST approach:

TU Dresden, 12th January 2015 An Update on Features Folie 25 von 51

assumption: corner’s intensity is offset from center

dominant direction ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗

TU Dresden, 12th January 2015 An Update on Features Folie 26 von 51

TU Dresden, 12th January 2015 An Update on Features Folie 27 von 51