With Seeing OPenCV

Library A Computer-Vision
comwith manystandard experiment free,oPen-source ,lPenCV - Intel's withouthaving putervision algorithms - can librarY t lcomputer-vision Theseinclude OpenCVis a f ree, oPen-source to code them yourself. computer-vision simplify Vgreatly ellipse detection, andcorner line, for C/C++ edge, a l p r o g r a m m i n gt. i n c l u d e s d v a n c e d computervision library for fitting,imagepyramids multiscale can downloadit - facedetection, track- programmers. You face capabilrties various matching, template processing, filtering, from http://sourceforge'net/proj Kalman ing,facerecognition, cosine, (Fourier, discrete transforms (Al) of anda variety artificial-intelligence ects/opencvlibrary. and transforms), more. anddistance of version the Intel released first form. methods in ready-to-use In addiit in computer- OpenCV 1999.Initially, required manybasic tion, it provides puter-visionmoduIes I That High-tevecom Library. lmageProcessing APls. Intel's via algorithms its lower-level vision high-level several includes OpenCV removed, was eventually dependency of A good understanding how In capabilities. addition to faceas get- and you can now use OPenCV a work is the keyto thesemethods it and recognition, tracking, detection, library. OpenCV standalone whenusing ting goodresults opticalflow (usingcamera lt is multi-platform. suP- includes you OpenCV l'll series, tntroduce In thisfive-part 3D structure), motion to determine and and Linux, oorts both Windows and to OpenCV showYouhow to uselt and calibration, stereo. camera With one MacOSX. recently, to implementface detection,face more (CVCAM, whichl'll describe Then, exception and face recognition. tracking, methods are Al and machine-learning its later in this article), interfaces to l'll take you behindthe scenes often applications Computer-vision independent. these methods platform explainhow each of or otherAl learning machine require worksand giveyou tips and tricksfor in are of Some these available methods. them. getting mostout of the package. Learning Machine OpenCV's OpenCV introduces firstarticle This O p e n C Vh a s s o m a n Yc a P a b i l i t i e s , get it andgiveyoua l'll tellyou how to and view for few pointers settingit up on your i t c a n s e e m o v e r w h e l m i n ga t f i r s t . lmagesampling transformations you'll need only a few to and Fortunately, how to read learn You'll comouter. a to It'softenuseful Processgroup convert g e t s t a r t e d . l ' l l w a l k Y o u t h r o u g h a video, capture files, writeimage includes as of pixels a unit. OpenCV pixel u s e f u ls u b s e ti n t h i s s e r i e s . and formats, access color between Here'sa summaryof the major interfaces for extracting image interfaces. OpenCV data- allthrough resizing, sampling, random in categories OpenCV, subregions, functionality just released w a r p i n g , r o t a t i n g , a n d a P P l Y i n g 1.0, version whichwas perspective effects. at the timeof thiswriting:

Overview OpenCV

Features

.!r

lmage and videoI/O let interfaces Youreadin These from files,or from live imagedata video feed. You can also create files. andvideo image and computer-vision General p ng mage- rocessi aIgorithms (mid-and lowJevd APls) Usingtheseinterfaces, can 1lou
manY 1. I FIGURE AmongOPenCV's lcapabilitiesarelace detection (top ; left), contour detection (top right) (bottom). I and edge detection

and analyzing Methodsfor creating I ) binary (two-vaued images used are images frequentlY Brnary that scan for systems in inspection A or defects countparts. binary shape when is alsoconvenient reoresentation to an locating object grasP. 3D Methodsfor comPuting information Thesefunctionsare useful for - either I mapping with and localization I from views rig a stereo or with multiple camera. I a sinole

62 srnvo01.eoo7

experimental code. However, the simplest interfaces for Graphics are These interfaces you write text face recognition let The In to anddrawon images. addition vari- in this module. possibilities, these codebehind themis ous fun and creative for functions useful labeling mark- specialized face for and are ing.Forexample, youwritea program recognition, and if that detects it's to objects, helpful label they'rewidelyused for thatpurpose. images theirsizes locations. with and Customizing Windows Install a For Windows users,OpenCVis installaeasy install, the default to and tion will work. But a bit of advance you happier planning with may leave Here theresults. area few suggestions. SinceOpenCVis a developers' toolkit - not a program- you may other want to locate it somewhere lf Files directory. than your Program you do preferto locateit elsewhere, you that before runthe installer, decide when asked. and enterthat location you I suggest alsodecide before to installing how youwant Windows find the OpenCV dlls.You can either PATH variable to modifyyour system's include theirlocation, youcanmove or from OpenCV's them.after installing, "bin" directory your SYSTEM_ROOT to directory. lf you prefer movethe dlls,but to aren'tsurewhereyourSYSTEM-ROOT directoryis, you can locate it by at running sysinfo the utilityavailable ities. www.cognotics.com/util Basiclnstall for is lf you preferto modifythe PATH OpenCV Linuxor MacOSX O p e n C V ' su n c t i o n a l i t iys c o n f packaged a source-code the dlls,you can archive. ratherthan moving modules. as tainedin several have the installer that for you do CXCORE containsbasicdatatype You'll need to build both staticand "Add bin the libraries. can either by selecting checkbox You For example,the data shared-object definitions. point,and rectan- build RPM firstandinstall fromthat. directory PATH." to an structures image, for gle are defined cxtypes.h. in CXCORE o r c o m p i l ea n d i n s t a l li t d i r e c t l y . for After Installing linear algebra statis- Instructions doing both are in and alsocontains The OpenCVdirectorycontains func- INSTALL. methods, the persistence tics The subdirectories. docsdirectoThe Windowsdownloadis pack- several tions,and error handlers. Somewhat for that installs ry contains htmldocumentation all functions draw- aged as an executable for oddly, graphics the functions datatypes. and here, well. OpenCVwhen you run it. lt places the OpenCV as ingon images located are is filesinto a directory your and OpenCV of Sincethe best documentation a CVcontains imageprocessing you yoursystem working want optionally modifies example, mightalso camera calibrationmethods. The choice, "samples" geometry the directory. the binaries, to browse functions are path to include OpenCV computational filters. By The headerfiles you'll need to and registers several DirectX alsolocated here. programs that whenyoucompile in doc- default, it installsto C:/Program include CVAUX describedOpenCV's is amongthe useOpenCV distributed are and Fi les/OpenCV/<version>. umentation containino as obsolete ML contains machine-learning interfaces Theremaining functionalityconis tained HighGUl CVCAM. in and Both of located a directory in named theseare "otherlibs," making to themeasy miss. HighGUl the l/O Since contains basic you'llwant to be sureyou interfaces, itl the don't overlook lt alsocontains multi-platform windowing capabilities. res ms Datastructu and algorith you can CVCAM containsinterfaces for With these interfaces, videoaccess throughDirectX 32-bit on store, search,save, and efficiently platforms. (also Windows HighG I However, U manipulate large lists, collections graphs, trees. videointerfaces. this In alsocontains called sets), and l'll in article, coveronly the interfaces They're to HighGUl. simpler use,and Data oersistence provide lf convenient they work on all platforms. you're These methods typesof usingWindowsXP or 2000,you may interfaces storingvarious for get a performange them later. boostby switching datato diskand retrieving to the CVCAM interfaces, but for Figure shows few examples 1 a of OpenCV the simpler onesin capabilities action:face learning in OpenCV's are contour detection, edge HiohGUl iustfine. and detection, detection. GUImethods OoenCV includes own windowits ing interfaces. Whiletheseare limited compared what can be done on to a each platform, they provide simple, API images, multi-platform to display acceptuser input via mouseor keycontrols. board, and implement slider

Math routines image processing, for computervision, and image interpretation includes mathcommonly OpenCV used,algorithms from linearalgebra, geometry. statistics, computational and

FIGURESelecting 9. header files OoenCV in Windows place to intoa single include directorv.

Organization

Installing OpenCV

SERVO O1.2oO7 63

1 //

Iugclo.

c rcrd rnd urlce rmgee

3 // Exarele showrng how lo a s llDcludc "cv.h" "hrghgur.h" a llncludt ? llncludc <ctdlo.h> 9 lns ulD (lnt r0 { 11 lplluoc. t3 la ls t 6 l? 18 19 21 22 23 2{ 7t 26 21 28 29 30 31 32 !rgc, char'r

FIGURE Example 3. program that reads an imagefrom a file and writes it to a second fife in a difterent compressionformat.

trqlv)

DIupIag " 9,

,// Lord aD r&Efc tron lrlc wlordlugc("n!'_rruge. rFg', CV-LoAD-IHAGI-ItNCaN|GED), r! ( !9InpIE) { ifall,ed Eo Load lnpul lMgc\n"),' lDrlDcl(tcdct!' rcBurD -1; l n.m, a drf lcrcnt /,/ ntl'.a Ehe rega to c frl,e ttth -- .Dng lntgaad of rMgc lorrut / / usrag a drllerent 1l( ldsrvclugc("nl'_lnage_coF\'.Fng". ) Plnpl4) { "fa1l,ed io *!lcc amge frle\i')i fprlnl!(grderr. l // Rercnber ro lree rug! wRel.aschrge(6plDplq), rrgurD 0; nrcly rlEer uslng 1E !

. JPg

*.hpp. Therewill be You lots of matches. don't need all of for them. Headers all modules except HighGUl are in "include" separate inside each directories Youcan skip module. the headers in "src" for directories these modules.For you'll need HighGUl, locatedin highgui.h, otherlibs/highgui.

path and the names the of the library Thestatic libraries to static libraries use. cv.lib, you needto linkto arecxcore.lib, for Later, facerecogniand highgui.lib. These tion,you'llalsolinkto cvaux.lib. "lib" in OpenCV's directory. are Reading and Writing lmages lmage l/O is easywith OpenCV. Figure3 showsa comPlete Program listing for readingan image f rom file, file and writingit as a second in a format. compression different call To readan imagefile, simply passing the filename it cvLoadlmage0, (line OpenCV most 14). supports common PNG, including JPEG, and image formats, format Youdon't needto provide BMP. determines information. cvLoadlmage0 the fileformatby reading file header. To write an image to file, call This cvSavelmageQ. function decides whichfile formatto usefrom the file the In extension. this example, exten"png," so it willwritethe image sionis format. datain PNG cvLoadlmagefl and Both cvSavelmagefl in the HighGUl are module. you're the finished using input When free from image received cvLoadlmagefl, as on it by calling cvReleaselmageQ, line of takes address a an 29. Thisfunction pointer its input because doesa it as "safe strucrelease." freesthe image lt it, After tureonlyif it'snon-null. freeing it pointer 0. to the sets image

the ton

//

cvRr

Cok

gra f ron ano Wor

two sou It w

you don't modules. Although OpenCV needto do this, I liketo gatherthem include directory. into together a single you and Windows, On both Linux the the by canlocate headers searching for install directory and subdirectories *.h, that filenames matchthe pattern
t 2 3 a t ? // // // // Crpcur..c Eraralc ahowlDg vidco frG! lrow co coDtect t to a wcbcs tnd

with Programming Basics Some OpenCV:

and More about Headers Libraries programs needto MostOpenCV Later, for cv.h include and highgui.h.

tma freq , call

oflg

Hov

caplura

llacluda llDcludc 0 alacludc t fiDcludc

'eidro.h" "srr1ng.h" ncv.h" "!:rghgur . h" r!gc, ' r chu " r!gr) . oi oi

!lln! t2( 13 l. 16 t6 18 19 20 22 23 2a 25 21 23 29 30 32 33 3l 35 36 3? 3 39 a0 tl a2 a3 aa a5 a6 a? t3 at

u1D(LDE CvcrDtule fDllugc ins chr!

pcrpEurc pvid.oP!@ li llleD.E[50];

vldco crptura // Inrtralrz! - cvclplurcfrotrllll pcrpEule lpcrDBurc tl( ! fprltrE!(9Bdc!!, -1; lcEula ) vl,deo // CaDEure tlrlcc !o:(1-0; 1<3, 1++) { Dvr,daoFre lDvld.oFre 1l( cvgqclyFre( ) 'farLed l!s9 "!arJ,ed

CV_CAP_IIY

),

cc lnrcr!Irze

Tldeo

caFtur?\n"1,'

aad x!l'!c

th.n

a5 llLes

!Capsu!!

);

fPllagl(gtdar!' 1

ic

gcl

a vrdcc

f:anc\r")i

vrdco trlG a! an lroge llle lhc cap!u!.d // ntLtc "vrdeoFrlRtd.tpg', 1.1); rD!1nE! (f1lene. (ttla!.c, pvldQofre) 1! ( lcvsrvcfugc ) ( "fal}ed is\r", io w.t!e lmage frle lDri'aE!(gEde!!, l // // l // tcr&inaEe wRclcarecapturc lrEuln 0; vrdco crplura ( EtfrDtule and !!ce ,, captula lelourcel fUPORTANT: Don'! froE cvQuclyFlcG(l rclcase ! or rcdlly ihc r@gc

lll'cntr),

..curncd

face recognition, sta we'llalsoinclude orig cvaux.h. The headremaining tron er files includare fiel ed by these headers. topJevel . W lf you'veleft the header files .H in multiple direc(defa lt Video Input u Live tories imageframesfrom a Capturing installation), your webcam, otherdigital videodevice, or ind maKe sure from file. is nearly easyas loading as exa c o m p i l e r ' s path con- Figure showsa completeprogram 4 cha include capframecapture, to tainsthesedirec- listing initialize 'n( video frames, tories. lf you've ture and store several gathered the interface. the four and close caoture on pte interfaceinitialized, is Thecapture headers into one cvCaptureFromcAMQ. inte include directory. line19,by calling pixe This functionreturnsa pointerto a make sure that You structure. won't access CvCaoture Gra directory is on you'll Instead, store y o u r c o m p i l e r ' s thisstructure directly. path. the pointer pass cvQueryFramefl. to to Acc include video using Whenyou'refinished Your linker to input, call cvReleaseCapture() will need both ope As with releasevideo resources. exa FIGURE Example cvReleaselmageQ,pass address 4. you the gra program that of the CvCapture pointer to seri live video captures prx cvReleaseCapture( frames and stores ). them as files. modifv Don'trelease otherwise or they

or.eooT 64 senvo

the lpllmageyou receive from cvQueryFrameQ! you need lf to modify image data, createa copy to work with:
//
lnl

Conv 'l.e
/m:^6 -n

'-ideo
md'L^l

f-ame
hrh^6 -

ers and other high-level constructs. pixel-level All calculations are performed inside OpenCV functions. However,if you write your own image-processing you may need algorithms, to access raw pixelvalues.Hereare two ways to do that:

cvclonelmage
/ [nqart rzorrr

(pVrdeoFrame ) ;
im:ao-nrnna<<inn nnr]o horo

//

l.YoF

rha

enmr:ftsa

cvReleaselmage

.r usrng rL ( &plmgTochange ) ;

1. SimplePixelAccess pixelsis with the The easiest way to read individual cvGet2D0 function:
CvScalar cr,Get2D(const CvArr*, int row, int col);

Color Conversions Figure showscode for converting color imageto 5 a grayscale. has OpenCV built-in support converting and for to from manyuseful colormodels, including RGB, HSVYCrCb, "The (For and CIELAB. a discussion color models, of see Worldof Color," SERVO Magazine, November 2005.) Notethat the conversion function, cvCvtColorQ, requires two images its inputlist.Thefirstone,pRGBlmg, the in is pGraylmg,thedestination image. second, source The is image. It willcontain conversion the result whencvCvtColoO returns. Because paradigm passing this of source destination and images a processing to function common OpenCV, is in you'll frequently needto create destination a image. line25, a On callto cvCreatelmageQ creates image same an the size the as pixel original, uninitialized data. with How OpenCVStoreslmages OpenCV stores images a C structure, as lpllmage. IPL standsfor lmage Processing Library, legacyf rom the a original OpenCV versions thatrrequired product. this Thelpllmage datatype defined CXCORE.addi is in In tion to raw pixel data,it contains number descriptive 1 / / a of 2 // fields, collectively the lmage called Header. These include 3 //
{ //

parameters: This function takes three a pointer a data to con(CVArr*), array tainer and indices row andcolumn for location. Thedatacontainer be an lpllmage can structure. topmost The rowof pixels row=0, the bottommostrow=height-1. is and is The cvGet2Dflfunction returns C structure, a CvScalar, defined as
r\modof cf F,-ts

CvScalar double val l4l ;

{
l

CvScalar;

The pixelvalues for each channel are in valIi].For grayscale pixelbrightness. other images, val[0]contains The threevalues set to 0. Fora three-channel, image, are BGR green:val[1and red=val[2]. blue:val[0], ], The complementary function, cvSet2D0, allowsyou to modifypixel values. defined lt's as
ConvelrToc:ay.c Exanple rhowtno to grayscale "sid:.. tr' "sir:rq.h' "cv.:." ":rg:aqer (1nt r r how !o coDverg an lroge lroB color

. Width- lmagewidth in pixels . Height lmage height pixels in . Depth - One of several predefined constants that indicate numberof bits per pixelper channel. the For example, depth:lPL_DEPTH_8U, for each pixel if data channel stored eight-bit, are as unsigned values.

6 lrncj,udc ? fiDcludc g tiDclude 9 llncluda t0 1l iDt. uiD 13 lr 16 17 l8 Ipllngc lpllmge

. hn chlrrr : rrgv) O. 0,

r!gc.

pRGBIq pcraylq

the RGB rBge / / Lo.d lron llle - flLordlDge("n}'_rege.jpg", pRGBIT{ rt ( TpRGBIE) { tprlDc!(3cdc!!, -l; rctuln 'faaled ro l,oad

CI/_LOAD_II|AGE_UNCHAIIGED}

19 . nChannelsThenumber datachannels (fromoneto 2 0 of 2f four). Each channel contains typeof ptxel one data. examFor ple,RGB green, blue 2 3 images three have channelsred, and 2a (These sometimes intensities. are called images, BGR because 29 pixeldata are storedas blue, green,then red values.) 26 2 ; Grayscale images contain onechannel pixel only brightness.2 E 29

rnpuc

rroge\nnli

Allocare // pcrayllg

rhe

glayccalc

rruEe pRcBtne->hc:ghr). pRcBfEg->depth, 1 t t

cvcreatclBge ( wsize(pRcBlne->w1dch,

Convetc rE Eo grrygcale // pcraylng, cvcvtcolor(pRGBIry, / / i!( { !p!tnt!(sgde!r. "falIed !c

CI/_RCB2GRAy)t f tl.e pcr.yl4) lnage flie\i")i )

Accessing PixelValues 3l It'spossible create to many types functionality of using 3 2 33 OpenCV without directlyaccessing pixeldata. For 3 { raw example, facedetection, the protracking, recognition and 31 gramsdescribed laterin this 38 series nevermanipulate raw FIGURE Example 39 5. pixel data directly.Instead, program converting { 0 for a colorimage grayscale.il 2 t to theywork with imagepoint4

Wrlre g:ayrcalc tre rruge ro a lcvsavaf@ge("n.'_rnage_graj..tFg", r:lEe

/,/ Ftec r@ge cvRclcrrcfDqc cvRcleasclugc return c,'

rcnolt' ( 6pRGBIE0) ,. (6pcraylDo) t

01.e007 SERVO 65

Sourceforgesite opcncvllbrary trttp:/ / sourcctots c.nd I Wolccls I

if your imagedata are needto castthe data type. Forexamp|e, input type),you,dcasteach bytes(the most common unsigned using'rt' or oth.erwise "a char* ,.iri ao unsigned '" "lilou;;accessing beforeassigning'

usergrouP-^ OfficialOPenCV grouq OpcnCV I http:/ /tcch. groupr'yatrooromI Wiki OpenCV nct lrttp://opcncvllbrary'courcctorgc' in listings thisarticleareavailable code for the Program Source | *nlo at wrrwr'cognotlcs'com/opcn1 for download
row' void cvSet2D(CvArr*, int Cvscalar) ; rnt cort

(single-channel) outt ttorn grlyscate is eightbits(onebyteper pixel)' and the datadepth imaoe, with yorid u...tt PixelIrow][col]
pixelIrow] Lcoll = ((uchar*) (PImg->r-mageData + aela*prmq-twidthstep + col) );

are values, interlaced' channel images' In multi-channel values: redpixel and green' to snippet "-tt" blue' a Here's code
qrpn - olmq->wjdthscep; nchan = Plmg->nCharulels; /l = 3 for a BGR image = Plmg->imageData; buf b l u e l r o w l I c o ] .I = ( (uchar*) (buf + row*widthsteP nChan*col); = ( (uchar*) (buf + row*widchsceP tt66u.'*69] + 1); +

-'' 2. FastPixel Access ul9 easy to use' and cvSet2D0 afihough cvGet2DQ ano more than a few pixelvalues' if you want to access directly matters,you'll want to read values oerformance imageData i.rn ,f.r.raw databuffer,lpllmage " :- -^.., In asa 1Darray' rowftuq. datarnthe bufferarestored

greenIrow] [col]

redIrow] lcoll + in values the second ( (uchar*) (buf + row*widthstep iitti, torfo*.d by pixel + 2); pixel data are aligned'and nchan*col '*tont' For performantt four-byte on row sothateach starts aneven if padded necessary, the indicates widthStep' eight bits (for n t..ona field, lpllmaqe ililil. if Finally, imagedepth is greater,than That data row'spixel startof each ihe needto,lt:ntft' multiple between youid. of number bytes + i*lpllmage'widthStep' .rurpi.,'ipL-oepin-:ii;, imageData the buffer offset by at iilr."* r staris lpllmage eacr' uatueanO'muttipty astypechar*'so youmay [vi.r"tor. is defined imagedepth.lt's very lpllmage.imageOata iti.-nrru"|. of data bytesfor your in which a that however, iou'll encounter situation unlikelv, dtrectty' pixel multi-byte values uo, tutt access

row listed values'llT.1':ti are alf r.j;; ;'A; ilat rs, pixel row' and so on'

ulnnlncROBOTT rreen tcRvot! tulTllotlT

HelP Finding
r."rp i"";r;;'LI r't!";laatthe ::::: :::fY ptr." rile'1f::root ir,"lilsralL
il;ffi.;;;v.
r':':9,Oj:::,Y' insta problems ||ing,o lf youhave :::::tJ '"t:l

?r-v"i:9fl:Y helpfulsetup and troubleshoottng contains il;;,-;lt; you maywant to yourquestion' toi ri ihesedon't answer usergroup Thesroup's Yahool official il;,; ;*" io the sidebar' in the Resources Unl it is in the docs/ref for APIdocumentation eachmodule except the one for manuals All reference ,rUjil.t.y htm' in the docsdirectory' from index are'linked AV;Ui

i:;:)'#J:iL5!i3 "::# P:fli[: ;:,t&;" ;i'i.'!*oto I:?lr'"": o ffi ;"

o:;:1"'

with ion. and steering

UP Coming ...
to detect faces with Next month,l'll show You how the interfaceBe behind the and OpenCV exPlain algorithm seeing I You

ry:I:d l'.si manv :: in control.Used i l"ii ;fi;;; tiiing'iorn dualspeed joystick n robotsSi gle uI com successf petitive ?p:r,t^*l:,:3

;::,:#;il.r[iJ,o'*""Y'^,::-:::^?.'l:T:li::ti:[; s' " di i.il ;';;;l'"il'" opoo'iterection

:ff ?:tY::: :':: v v ' r r v - - / proportronal,. ' PUJ' ;;iiilL ' v " Jlomptetetvi r n i l u tr a d il':s:: p llf,-'*::ffi"::o o Com atiblewith Syr your Futaba,JR,Hitec,o t . t "izac arrailahrl. rt^^! ^m^ l::::"']i*': o i""' vu'rusvo und uTp#;;;tir'i)ui
above' 55v 75A per motorunit pictured i'n" ndrn+ze www.vantec.com

lfr

sle and robotics' las wo*ed asa ;;:;:,:;";';;;;;;;;;i"; and b;aow' at Evotution,Robotics A;;:;;;r;t";-xs,nii group at UC research ""'oipii'-ution of is a member 5o(3), ';;;;;;. an or sodavistoa sh; is-ii"'i'tii oisinat devetopers san Diego' tytt"' at lJc experimental ro(;"''""ognii6n i[aavision wasbuittwtthoPnCV'

01.eoo7 66 sgnvo