Tutorial: OpenCV haartraining (Rapid Object Detection With A Cascade of Boosted Classifiers Based ...

Pagina 1

Naotoshi Seo
Tutorial: OpenCV haartraining (Rapid Object Detection With A
Cascade of Boosted Classifiers Based on Haar-like Features)
First Edition. 03/12/2006
Last Modified. 06/05/2007
Tag: SciSoftware FaceDetection ComputerVision OpenCV

Objective
The OpenCV library gives us a greatly interesting demo for a face detection. Furthermore, it provides us
programs (or functions) which they used to train classifiers for their face detection system (called HaarTraining).
Thus, we can create our own object classifiers using the functions. It is interesting.

However, I could not follow how OpenCV developers performed the haartrainig for their face detection system
exactly because they did not provide us several information such as what images they used for training. The
objective of this report is to provide step-by-step procedures for following people.

My working environment is Visual Studio + cygwin on Windows XP, or on Linux. The cygwin is required
because I use several Linux commands.

A picture from the OpenCV website

Data Prepartion
Positive (Face) Images

I downloaded from The UMIST Face Database.

FYI: There are database lists on Face Recognition Homepage - Databases. and Computer Vision Test Images.

Negative (Background) Images

Kuranov et. al. [3] states as they used 3000 negative images.

http://note.sonots.com/?cmd=rss 21/01/2008 08.34.44

Tutorial: OpenCV haartraining (Rapid Object Detection With A Cascade of Boosted Classifiers Based ... Pagina 2

Fortunately, I found http://face.urtho.net/ (Negatives sets, Set 1 - Various negatives ) which has about 3500
images. But, this collection was used for eye detection, and includes some faces in some pictures. Therefore, I
deleted all suspicious images which looked including faces. About 3000 images were remained, thus they should
be enough.

The collection is available at the Download section (But, it may take forever to download.)

Create Samples (Increase Images)

We can create traning samples and testing samples with the createsamples software.

createsamples
The createsamples software has mainly four functionalities.

1. Create training samples (a vec file) from one image applying distortions
2. Create training samples (a vec file) from some images without applying distortions
3. Create test samples (images) and their ground truth (a description file) from single image applying
distortions
4. Show images within a vec file

I describe these functionalities here because the Tutorial [1] does not explain them clearly (but please see the
Tutorial [1] also for further options).

1. Create training samples from one image applying distortions.

This function is launched when options, -img, -bg, and -vec were specified.

-img [a positive image]

-bg [a description file of negatives]
-vec [name of the output file containing the generated samples]

For example,

$ createsamples -img face.png -num 10 -bg negatives.dat -vec samples.vec -maxxangle 0.6 -
maxyangle 0 -maxzangle 0.3 -maxidev 100 -bgcolor 0 -bgthresh 0 -w 20 -h 20

Only the first <num> negative images in the <description file of negatives> are used.

2. Create training samples from some images without applying distortions

You may think this function is a simple file format conversion function.

This is launched when options, -info, and -vec were specified.

-info [a description file of positives]

-vec [name of the output file containing the generated samples]

In this case, the option -num is used only to reduce the number of samples to generate, this does not generate
many samples from one image applying distortions.

For example,

http://note.sonots.com/index.php?SciSoftware%2Fhaartraining 21/01/2008 08.34.44

Tutorial: OpenCV haartraining (Rapid Object Detection With A Cascade of Boosted Classifiers Based ... Pagina 3

$ createsamples -info samples.dat -vec samples.vec -w 20 -h 20

3. Create test samples and their ground truth from single image applying distortions

This is triggered when options, -img, -bg, and -info were specified.

-img [a positive image]

-bg [a description file of negatives]
-info [a generated description file for the generated test iamges]

In this case, -w and -h are used to determine the minimal size of positives to be embeded in the test images.

$ createsamples -img face.png -num 10 -bg negatives.dat -info test.dat -maxxangle 0.6 -
maxyangle 0 -maxzangle 0.3 -maxidev 100 -bgcolor 0 -bgthresh 0

This generates tons of jpg files such as

The output file name format is as <number>_<x>_<y>_<width>_<height>.jpg, where x, y, width and height are
the coordinates of placed object bounding rectangle.

Only the first <num> negative images in the <description file of negatives> are used.

4. Show images within a vec file

This is triggered when only an option, -vec, was specified (no -info, -img, -bg). For example,

$ createsamples -vec samples.vec -w 20 -h 20

Create Description File

The format of a description file which is used by createsamples is [1]

[filename]
[filename]
[filename]

http://note.sonots.com/index.php?SciSoftware%2Fhaartraining 21/01/2008 08.34.44

Tutorial: OpenCV haartraining (Rapid Object Detection With A Cascade of Boosted Classifiers Based ... Pagina 4

.
.

such as

img/img1.jpg
img/img2.jpg

or if it is used for testing,

[filename] [# of objects] [[x y width height] [... for 2nd object] ...]
[filename] [# of objects] [[x y width height] [... for 2nd object] ...]
[filename] [# of objects] [[x y width height] [... for 2nd object] ...]
.
.

such as

img/img1.jpg 1 140 100 45 45

img/img2.jpg 2 100 200 50 50 50 30 25 25

The first format can be easily created with the find command as

$ cd [your working directory]

$ find [image dir] -name '*.[image ext]' > [description file]

such as

$ find ../../data/negatives/ -name '*.jpg' > negatives.dat

$ find ../../data/umist_cropped/ -name '*.pgm' > positives.dat

EXTRA: random seed

The createsamples software applys the same sequence of distortions for each image. We may want to apply the
different sequence of distortions for each image because, otherwise, our resulting detection may work only for
specific distortions.

This can be done by modifying createsamples slightly as:

Add below in the top

#include<time.h>

Add below in the main function

srand(time(NULL));

The modified source code is available at svn:createsamples.cpp

http://note.sonots.com/index.php?SciSoftware%2Fhaartraining 21/01/2008 08.34.44

Tutorial: OpenCV haartraining (Rapid Object Detection With A Cascade of Boosted Classifiers Based ... Pagina 5

Create Training Samples

Kuranov et. al. [3] mentions as they used 5000 positive frontal face patterns and 3000 negatives for training, and
5000 positive frontal face patterns were derived from 1000 original faces.

However, you may have noticed that none of 4 functionalities of createsample software provide us the
functionality to generate 5000 positives from 1000 images at burst. We have to use the 1st functionality of the
createsample to generate 5 (or some) positives form 1 image, repeat the procedures 1000 (or some) times, and
finally merge the generated output vec files.

I wrote a program, mergevec.cpp, to merge vec files.

I also wrote a script, createtrainsamples.pl, to repeat the procedures 1000 (or some) times. Please modify the path
to createsamples and its option parameters directly written in the file. I specified 7000 instead of 5000 because the
Tutorial [1] states as "the reasonable number of positive samples is 7000."

The input format of createtrainsamples.pl is

$ perl createtrainsamples.pl <positives.dat> <negatives.dat> <vec_out_dir>

And, the input format of mergevec is

$ mergevec <vec_collection_file_name> <output_vec_file_name>

Example)

$ perl createtrainsamples.pl positives.dat negatives.dat samples

$ find samples/ -name '*.vec' > samples.dat # to create a collection file for vec files
$ mergevec samples.dat samples.vec
$ createsamples -vec samples.vec -show -w 20 -h 20 # Extra: If you want to see inside

Create Testing Samples

Testing samples are images which include positives in negative images and locations of positives are known in the
images. We can use the 3rd functionality of createsamples to do it. But, we can specify only one image using it,
thus, creating a script to repeat the procedure would help us. The script is available at svn:createtestsamples.pl.
Please modify the path to createsamples and its option parameters directly in the file.

The input format of the createtestsamples.pl is as

$ perl createtestsamples.pl <positives.dat> <negatives.dat> <output_dir>

This generates lots of jpg files and info.dat. The jpg file name format is as
<number>_<x>_<y>_<width>_<height>.jpg, where x, y, width and height are the coordinates of placed object
bounding rectangle.

Example)

$ perl createtestsamples.pl positives.dat negatives.dat tests

$ find tests/ -name 'info.dat' -exec cat \{\} \; > tests.dat # merge info files

http://note.sonots.com/index.php?SciSoftware%2Fhaartraining 21/01/2008 08.34.44

Tutorial: OpenCV haartraining (Rapid Object Detection With A Cascade of Boosted Classifiers Based ... Pagina 6

$ # find tests/ -name 'info.dat' | xargs cat > tests.dat # should be faster, but may not
work

Training
Training is done by the haartraining software.

haartraining
Kuranov et. al. [3] states as 20x20 of sample size achieved the highest hit rate. Furthermore, they states as "For
18x18 four split nodes performed best, while for 20x20 two nodes were slightly better. The difference between
weak tree classifiers with 2, 3 or 4 split nodes is smaller than their superiority with respect to stumps."

Furthermore, there was a description as "20 stages were trained. Assuming that my test set is representative for
the learning task, I can expect a false alarm rate about and a hit rate about
."

Therefore, I used 20x20 of sample size with nsplit = 2, nstages = 20, minhitrate = 0.9999, maxfalselarm = 0.5
such as

$ haartraining -data trainout -vec samples.vec -bg negatives.dat -nstages 20 -nsplits 2 -

minhitrate 0.999 -maxfalsealarm 0.5 -npos 7000 -neg 2984 -w 20 -h 20 -mem 512 -mode ALL

The "-mode ALL" uses Extended Sets of Haar-like Features [2]. Default is BASIC and it uses only upright
features, while ALL uses the full set of upright and 45 degree rotated feature set [1].

The "-mem 512" is the available memory in MB for precalculation [1]. Default is 200MB, so increase if more
memory is available. We should not use all system RAM because otherwise it will result in a considerable
training slow down.

#If you use OpenMP (multi-processing) supporting compilers such as Interl C++ compiler and MS Visual Studio
8.0, the haartraining is automatically built with the OpenMP. This is effective when your computer has multi-
CPUs.

#One training took two days.

Generate xml
The haartraing software did not generate a cascade.xml used by the facedetect software.

Although the Tutorial [1] does not mention how to generate a cascade.xml, OpenCV has a software to convert a
haartraining output dir tree into a xml file at the OpenCV/samples/c/convert_cascade.c (that is, in your installation
directory). Compile it.

The input format is as

$ convert_cascade --size="<sample_width>x<sampe_height>" <haartraining_ouput_dir> <ouput_

file>

Example)

http://note.sonots.com/index.php?SciSoftware%2Fhaartraining 21/01/2008 08.34.44

Tutorial: OpenCV haartraining (Rapid Object Detection With A Cascade of Boosted Classifiers Based ... Pagina 7

$ convert_cascade --size="20x20" trainout cascade.xml

Testing
under construction. I am not achieving sufficient results....

performance
We will evaluate the performance of the generated classifier using the performance software.

The input format of the software is

$ performance -data <haartraining_output_dir> -info <description_file_for_test_samples>

such as

$ performance -data trainout -info tests.dat

The output format is as

+================================+======+======+======+
| File Name | Hits |Missed| False|
+================================+======+======+======+

'Hits' shows the # of correctly found objects, and 'Missed' shows the # of missed objects (there must exist but not
found, as known as false negatives), and 'False' shows the # of false alarm (there must not exist but found, as
known as false positives).

facedetect
Fun with a USB camera or some image files.

$ facedetect --cascade=<xml_file> [<imagefile_or_videofile_or_cameraid>]

Discussion
The generated classifier did not work well. Specializing classifiers for only the frontal faces may achieve the
better result (UMIST images include 90 degree faces for each person.)

Download
The files are available at http://opensvn.csie.org/sonots/SciSoftware/haartraining/ .

HaarTraining Source Code, Binary, Makefile (VC++ Project File)

data The collected Image Datasets
result Generated Files

http://note.sonots.com/index.php?SciSoftware%2Fhaartraining 21/01/2008 08.34.44

Tutorial: OpenCV haartraining (Rapid Object Detection With A Cascade of Boosted Classifiers Based ... Pagina 8

This is a svn repository, thus you can download files at burst if you have a svn client. (I will not explain how to
install or use the svn client here.) Downloading Image Datasets may take forever...

The list of my additional utilities (I put them in HaarTraining/ directory):

mergevec.cpp - further details

vec2img.cpp - further details
createtrainsamples.pl
createtestsamples.pl

The following additional utilities can be obtained from OpenCV/samples/c in your OpenCV install directory (I
also put them in HaarTraining/ directory).

convert_cascade.c
facedetect.c

References
[1] Rapid Object Detection With A Cascade of Boosted Classifiers Based on Haar-like Features - OpenCV
haartraining Tutorial (This can be obtained from OpenCV/apps/HaarTraining/doc on your OpenCV install
directory).

[2] Rainer Lienhart and Jochen Maydt. An Extended Set of Haar-like Features for Rapid Object Detection.
Submitted to ICIP2002. ( http://www.lienhart.de/ICIP2002.pdf )

[3] Alexander Kuranov, Rainer Lienhart, and Vadim Pisarevsky. An Empirical Analysis of Boosting
Algorithms for Rapid Objects With an Extended Set of Haar-like Features. Intel Technical Report MRL-TR-
July02-01, 2002. ( http://www.lienhart.de/Publications/DAGM2003.pdf )

OpenCV haartraining (Rapid Object Detection With A Cascade of Boosted Classifiers Based on Haar-like
Features) - Naotoshi Seo
Learning-Based Computer Vision with Intel's Open Source Computer Vision Library
How-to build a cascade of boosted classifiers based on haar like features [pdf]
Empirical Analysis of Detection Cascades of Boosted Classifiers for Rapid Object Detection [pdf]
5000 positive frontal face patterns and 3000 negative. 5000 positive frontal face patterns were derived
from 1000 original face patterns by random rotation, scaling, mirroring, shifting. 20x20 of sample size with
nsplit = 2. nstages = 20
OpenCV for Linux　Haar-like 特徴を用いる高速物体検知
OpenCV: オブジェクト検出器関数
http://ai-www.naist.jp/people/kohei-t/OpenCV/reference/ref/OpenCVRef_Experimental.htm 和訳
An Extended Set of Haar-like Features for Rapid Object Detection [pdf]
MERL – TR2003-096 – Fast Multi-view Face Detection
Haar状特徴に基づくブースト分類器のカスケードを利用する高速物体検知
haartraining Tutorial 和訳
Object Detection Using Haar-like Features with Cascade of Boosted Classifiers
Manual to create cascade xml file. Same with the file located in OpenCV directory.
OpenCV: Object Detection Functions
FaceDetection - OpenCV Library Wiki
Boosting Haar-like features. The basic classifiers are decision-tree classifiers with at least 2 leaves.

http://note.sonots.com/index.php?SciSoftware%2Fhaartraining 21/01/2008 08.34.44