Automatic License Plate Location Using Template Matching

University of Wisconsin - Madison

ECE 533 Image Processing Fall 2004

Kerry Widder

Introduction
The task of recognizing some object in an image is one of the most difficult in the field of digital image processing. Many factors contribute to this difficulty, including variations in lighting and contrast, differences in orientation or size, variations in the surroundings of the object and other objects covering part of the object. This is also true for the specific recognition task of automatically finding a license plate in the image of a vehicle. The color of the vehicle may be the same as the plate, making the boundary between the plate and the vehicle difficult to detect. The vehicle may have other rectangular shapes such as lights, grills or bumpers that make it harder to distinguish the license from other parts of the vehicle. The plate may also be surrounded by a holder that covers part of it and alters its proportions. The plates vary greatly in size, color and layout from country to country. Some have extra symbols or pictures in the background that make it harder to identify the characters on the plate. The quantity of the characters may vary greatly, and some have all letters, others have all numerals, and the rest have some mixture of the two. The automatic identification of license plates provides benefits in many different applications. Tollbooth violators can be photographed and a ticket issued automatically based on the image of the vehicles license. Law enforcement officials can use it to identify speed limit violators based on images taken by cameras mounted in remote locations, or to automatically catch vehicles that run a red light. The operators of a parking structure can take an image of a vehicle when it enters and again at exit and correlate the ticket number with the vehicle to prevent fraud. Human operators reading the images and taking the appropriate actions could handle these applications. However, many organizations dont have enough personnel to handle these tasks manually. It is also more efficient and less costly to handle these tasks automatically. These factors make the automatic detection of license plates a topic that has generated a lot of interest. This project will attempt to find the location of the license plates in images of vehicles using template matching.

Approach
The task of automatically locating license plates in images has been approached in many different ways. One approach utilizes distance sets, a form of local descriptor that uses the spatial arrangement of features to identify an object [1]. Another uses morphological operations, such as smoothing, closing, opening and threshold, to attempt to isolate the features of a license plate [2]. The high contrast between the characters and the plate is exploited in another method that searches for the greatest gray-level contrast in an area [3]. Another method finds rows in the image with the most light to dark transitions, then

examines the columns in each row for the highest concentration of dark pixels [4]. Genetic algorithms have also been used in the decision making process of identifying the license plates [5]. This project will explore two mechanisms utilizing template matching, correlation and moment invariants. The correlation between two functions (or images) f(x,y) and h(x,y) is defined as f ( x, y ) h ( x, y ) = 1 MN
M 1 N 1 m =0 n =0

f * (m, n)h( x + m, y + n)

(1)

where f* denotes the complex conjugate of f [6]. Since the image functions dealt with in this project are real functions, f* = f. The results of applying the correlation function can be displayed as an image the size of f (assuming the size of f is greater than or equal to the size of h) with brighter pixels indicating a closer match between f and h at those points. If h is a template of the object to be found in image f, then applying the correlation function between f and h will result in image g(x,y), where the brightest spot will indicate the best match in image f of template h. Thus, correlation provides a simple way to search for the presence of a given object in an image. A sample license plate will be isolated from its vehicle image and used as the template. Each vehicle image will be correlated with the template to attempt to find the license in the image. The moment invariants of an image are statistical descriptors of the image that are invariant to translation, rotation and scale changes [6]. A set of seven of them is defined by
1 = 20 + 02
2 2 = ( 20 02 ) + 411 2

(2) (3)
2

3 = ( 30 312 ) + ( 3 21 03 )
2

(4)

4 = ( 30 + 12 ) + ( 21 + 03 )
2

5 = ( 30 3 12 )( 30 + 12 ) ( 30 + 12 ) 3( 21 + 03 ) +
2 2

( 3 21 03 )( 21 + 03 ) [3( 30 + 12 ) 2 ( 21 + 03 ) 2 ]

(5) (6)

6 = ( 20 02 ) ( 30 + 12 ) ( 21 + 03 ) + 4 11 ( 30 + 12 )( 21 + 03 )
2 2

(7)
2

7 = ( 3 21 03 )( 30 + 12 ) ( 30 + 12 ) 3( 21 + 03 ) +
2

( 3 12 30 )( 21 + 03 ) [3( 30 + 12 ) 2 ( 21 + 03 ) 2 ]
pq
00

(8)

where

pq =

(9)

and
=
p+q +1 2

(10)

for p + q = 2,3,. and where

pq = x x
x y

) ( y y)
p

f ( x, y )

(11)

and
x= m10 m00

and

y=

m01 m00

(12)

and
m pq =

x p y q f ( x, y )dxdy

(13)

Since images of vehicle license plates will be of varying size and orientation (unless the images are taken in a very controlled setting), the immunity of these descriptors to variations in orientation and scale makes them an attractive candidate for this task. The moment invariants of the template will be calculated and then compared with the calculated moment invariants of each object in the image. The Euclidean distance, defined as
D j ( x ) = x m j

for j = 1, 2,, W

(14)

(where W is the number of objects) will be used as the measure of the closeness of the match between each object and the template, with the minimum value considered to be the best match. Each image must be segmented into objects before the moment invariants can be calculated. The segmentation will be performed by first finding the edges using a Sobel operator, defined by the masks -1 -2 -1 0 1 0 2 0 1 -1 0 -2 0 -1 0 1 2 1

Figure 1 Sobel operator masks.

and then calculating the exterior boundaries of each object in the resulting edge image. The boundaries will then be processed to define a sub-image enclosing each object. The moment invariants will be calculated for each sub-image and compared to the template. The algorithm will propose the boundary image with the smallest distance to the template as the license plate match.

Work Performed
A set of ten vehicle images was collected. The images were 640 x 480 pixels. Care was taken to keep the license plate size relatively consistent to keep the task more manageable. One of the sample images is shown in figure 2. The images were converted from RGB to gray scale. One of the license plates was isolated, cropped and stored for use as the template, as seen in figure 3.

Figure 2 Sample vehicle image.

(a)

(b)

Figure 3 a) License plate template, b) license plate correlation template

The image processing was done using MATLAB [7]. Several MATLAB functions were used that came from the book Digital Image Processing Using MATLAB (DIPUM) [8]. The high-level algorithm is given in figure 4. A sample of the output is shown in figure 5. ________________________________________________________ Load template and vehicle image Create Gaussian filters for template and image Perform Gaussian filtering on template and image Perform Sobel filtering on filtered image and template to find edges Perform correlation between template and image using edge images Load correlation template Perform correlation between correlation template and correlation image Locate both correlation results in copy of original image and store Calculate moment invariants of original template Convert Sobel of image to binary for boundary location Perform opening to reduce stray protrusions Locate boundaries of objects in the binary image Process each boundary to find rectangle enclosing it For each such image, calculate moment invariants and store Calculate norm with template moment invariants and store Find best match Create sub-image of best match, store and display _________________________________________________________
Figure 4 License Plate Recognition Algorithm

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Figure 5 Results for one vehicle: a) Original image, b) Results of Sobel operator, c) Correlation with template, d) Correlation on correlation, e) Results of correlation methods, f) Image of calculated boundaries, g) Result of moment invariant method.

Results
The template matching by correlation method was successful for six of the ten vehicle images (60%), with one other that was close and three that completely missed. The correlation images had a distinctive cross-shaped pattern where the license plate was (see fig. 5(c)). In an effort to capitalize on that and to try to get better results, a template was created from the correlation of the original template with the image it was taken from, and this correlation template was then used as the template for doing a correlation between it and each correlation result (correlation squared). The results of this correlation on the correlation were not as good as the original correlation results, with success for only two images and three others that were close. A successful correlation is shown in figure 6, where the square target shows the upper left corner of the plate as determined by the correlation on correlation and the rectangular target shows the upper left corner of the plate as determined by the correlation results. The grill and headlight area of this vehicle yields high readings, but the plate provided a better match in this case. Two unsuccessful results are shown in figure 7. The first example shows higher values for the grill area than the license plate and the distinctive cross-shaped pattern is not evident by the license plate. In the second example, the crossshaped pattern is there, but the grill and headlight area yields higher readings.

(a)

(b)

(c)

Figure 6 Successful correlation result: a) original image with correlation matches indicated, b) correlation image, c) correlation on correlation image.

(a)

(b)

(c)

(d)

Figure 7 Unsuccessful correlation results: a) & c) original images with correlation matches indicated, b) & d) correlation images.

The moment invariant method resulted in the successful location of the license plate for only two of ten images (20%). Six of the eight failures were due to problems in getting a good boundary of the plate, the other two had enough intact boundary to select the plate as one of the objects, but the moment invariant measure failed to select the plates. The comparison to the template is done by taking the minimum and maximum x and y values of each boundary and using them to define a sub-image of the original image. These sub-images are then compared to the template. Therefore, if at least two full connected edges of a plate result from the boundary selection process, the plate will be one of the candidate objects.The first set of images in figure 8 shows a successful match. The plate boundary is almost complete. This ensures that the plate will be one of the objects compared to the template. The second set shows an image where the boundary image has two complete, connected sides of the plate, which will result in the complete license plate being an object to be compared with the template. However, the comparison failed to correctly select the license plate. The third set of images shows a case where the boundary of the plate is insufficient to select the plate as an object. In this case, the largest intact section of the plate boundary is one of the vertical edges. In the preliminary work on this project, this image had significant protrusions off the boundary of the license plate. Adding the morphological opening step in the process removed these protrusions in this case, but resulted in the boundary of the plate being lost. The fourth set shows a case where the boundary of the plate has a significant protrusion caused by an edge of the bumper that blends into the plate. This protrusion will result in the license plate being part of a larger object for comparison with the template. Even if this region had the closest match, it still would not be an exact match because of the extra area selected with the license plate. Finally, an example of a license holder complicating the process is seen in the final set of images. The holder in this image has letters at the top and blocks the bottom of the plate. The region selected by the boundary will include the letters on the top, but not the bottom of the holder. This region was not successfully matched to the template.

(a)

(b)

10

(c)

(d)

(e)

(f)

(g)

(h)

11

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Figure 8 Moment invariant results: a), c), e), g) & i) Original images, b), d), f), h) & j) boundary images illustrating various conditions, k) o) best match assigned by the algorithm for each respective image.

Discussion
The correlation method of template matching was moderately successful, but not reliable enough for commercial use. One difficulty with this method is creating a generic template. This work just used a sample plate as the template. Careful creation of the template may produce better results and would be an area for further study. This method also would probably be less successful if the images had greater variation in size or orientation of the plates. In general, correlation does not seem to be a good candidate for a robust license plate location method. In the moment invariant method of template matching, the identification of the object boundaries proved to be troublesome. Initially, some of the edge images had protrusions off the plates caused by nearby features of the vehicles, as seen in figure 8(h). The morphological opening step was added to try to eliminate these protrusions. This added step was partially successful at doing so, but it had the side affect of also sometimes chopping up the boundary of the license plate and making it impossible to select it as a region of interest for the moment invariant test, as seen in figure 8(f). Also, variations in contrast levels and the sharpness of the edges in the images made the task of applying a threshold to the edge image to convert it to a binary image difficult to do reliably for all cases. If the threshold is too low, there will be too many edges and the boundaries will run together, making the object separation difficult. If the threshold is too high, important features may be missing and the object of interest may not be identifiable. 12

The sample size for this project was not very big. To get numbers that would be statistically meaningful, more data would need to be collected. Further work could be done in several areas. The refinement of the template could have an impact on the success of these methods. The use of other methods of edge detection could be explored, which could especially benefit the moment invariant method, where difficulties were encountered in isolating the boundaries of the license plates. The many variations in the images make the location task difficult to achieve with simple template matching. To get better results, some type of adaptive method may be necessary to handle the wide variations in vehicle images.

13

References
[1] C. Grigorescu and N. Petkov, Distance Sets for Shape Filters and Shape Recognition, IEEE Trans. Image Processing, vol. 12, pp. 1274-1286, Oct. 2003. [2] J. Hsieh, S. Yu and Y. Chen, Morphology-based license plate detection in images of differently illuminated and oriented cars, J. Electronic Imaging, vol. 11(4), pp. 507-516, Oct. 2002. [3] T. Kula and J. Cicero, Image Processing Experiments, in Proc. 21st SIGCSE Tech. Symp. On Computer Science Education, Washington, D.C., United States, 1990, pp.167-170. [4] N. Ketelaars, Final Project: Automated License Plate Recognition, AIMe Magazine, vol. 1, pp. 9-12, 2002. [5] J. Parker and P. Federl, An Approach to License Plate Recognition, Computer Science Technical Reports, Dept. Computer Science, University of Calgary, Calgary, Alberta Canada, Oct. 1, 1996. [6] R. Gonzalez and R. Woods, Digital Image Processing, 2nd ed., Prentice-Hall, New Jersey, 2001. [7] MATLAB 7.0, The Mathworks, Inc., Natick, MA, 2004. [8] R. Gonzalez, R. Woods and S. Eddins, Digital Image Processing Using MATLAB, Pearson Education, Inc., New Jersey, 2004.

14

Appendix
MATLAB m-file listing
%UW Madison Dept. of Electrical and Computer Engineering %ECE-533 Image Processing %Fall 2004 Project %Kerry Widder % %License Plate Recognition % %This program attempts to find a license plate in the image of a %vehicle through two approaches: % 1. Correlation between a license template and the image. % 2. Create a binary image from the Sobel filtered image using a % % % % % % %This program utilizes Matlab 7.0. Some of the functions used came %from the book "Digital Image Processing Using Matlab", Gonzalez, %Woods, Eddins, Pearson Education, Inc., 2004. These functions are available %from the book's web site in the form of p-files. They will be %identified when first used by a comment '%DIPUM'. % %The program assumes a gray-scale image file 'car_input.bmp' and % a template license plate file 'template.bmp' and a correlation % template file 'corr_template.bmp' residing in the % working directory of the Matlab session. The output will be the %following files: % template_gaus.bmp % template_sobel.bmp % car_gaus.bmp % car_sobel.bmp - gaussian LP filtered template - Sobel filtered template for edge detection - gaussian LP filtered image - Sobel filtered image for edge detection threshold value, calculate the boundaries of the objects in the image, calculate the invariant moments of each object's area in the image and compare to the invariant moments of the template, using the smallest difference as the determination of the best match.

15

% car_correl.bmp % car_corr2.bmp % car_corresult.bmp % boundaries.bmp % lic_match.bmp % % clear all % load image

- correlation of image with template - correlation of correlation image with template - image showing correlation matches - image showing the calculated boundaries - image showing the object in the image that

has the best match to the invariant moments of the template

tem=imread('template.bmp'); fim=imread('car_input.bmp'); % [m,n]=size(fim); mp=2*m; np=2*n; Do=150; %cutoff for gaussian filter - image [tm,tn]=size(tem); tmp=2*tm; tnp=2*tn; Dot=15; %cutoff for gaussian filter - template % % perform gaussian low pass filtering to reduce detail of image %lpfilter() from DIPUM Hgaus=lpfilter('gaussian',mp,np,Do); %dftfilt() from DIPUM fimg1=dftfilt(im2double(fim),Hgaus); fimg=im2uint8(mat2gray(fimg1)); imwrite(fimg,'car_gaus.bmp'); % %perform gaussian low pass filtering to reduce detail of template Hgaust=lpfilter('gaussian',tmp,tnp,Dot); temg1=dftfilt(im2double(tem),Hgaust); temg=im2uint8(mat2gray(temg1)); imwrite(temg,'template_gaus.bmp'); % % perform sobel filtering to find edges of image

16

sobel1=[-1,-2,-1;0,0,0;1,2,1]; sobel2=[-1,0,1;-2,0,2;-1,0,1]; fsob1=imfilter(fimg1,sobel1,'conv','symmetric'); fsob2=imfilter(fimg1,sobel2,'conv','symmetric'); f2sob=fsob1.^2+(fsob2.^2); fimsob=sqrt(f2sob); fimsobel=im2uint8(mat2gray(fimsob)); figure(1),clf,colormap('gray'); subplot(2,3,1); imshow(fimsobel); imwrite(fimsobel,'car_sobel.bmp'); % % perform sobel filtering to find edges of template tsob1=imfilter(temg1,sobel1,'conv','symmetric'); tsob2=imfilter(temg1,sobel2,'conv','symmetric'); t2sob=tsob1.^2+(tsob2.^2); temsob=sqrt(t2sob); temsobel=im2uint8(mat2gray(temsob)); imwrite(temsobel,'template_sobel.bmp'); % %perform correlation between the image and the template %dftcorr() from DIPUM fcor=dftcorr(fimsob,temsob); fcorr=im2uint8(mat2gray(fcor)); subplot(2,3,2); imshow(fcorr); imwrite(fcorr,'car_correl.bmp'); %% %%perform correlation between the correlation and the correlation template %%load correlation template temcor=imread('corr_template.bmp'); fcor2=dftcorr(fcor,im2double(temcor)); fcorr2=im2uint8(mat2gray(fcor2)); subplot(2,3,3); imshow(fcorr2); imwrite(fcorr2,'car_corr2.bmp'); %%

17

%find the pixel of highest intensity in the correlation %and mark that spot in a new copy of the image %will be the top left corner of the plate % %find maximum intensity location %vector of max value in each column Colmax=max(fcor); %Col will be index of column containing max value [Max,Col1]=max(Colmax); %create vector containing column with max value Maxcol=fcor(:,Col1); %Row is row index of max value [Max2,Row1]=max(Maxcol); %translate Row=Row1; Col=Col1; %create 'target' to identify this location in a copy of the image % This target is a 6 x 20 pixel rectangle with the top right quarter and % the bottom left quarter white, the rest black target=[0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255; 0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255; 0,0,0,0,0,0,0,0,0,0,255,255,255,255,255,255,255,255,255,255; 255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0; 255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0; 255,255,255,255,255,255,255,255,255,255,0,0,0,0,0,0,0,0,0,0]; %% %find the pixel of highest intensity in the correlation of correlation %and mark that spot in a new copy of the image - must translate by 1/2 %the size of the correlation template, which is 59,123 % %find maximum intensity location %vector of max value in each column Colmax=max(fcor2); %Col will be index of column containing max value [Max,Col1]=max(Colmax); %create vector containing column with max value Maxcol=fcor2(:,Col1);

18

%Row is row index of max value [Max2,Row1]=max(Maxcol); %translate Row2=Row1+59; Col2=Col1+123; %create 'target2' to identify this location in a copy of the image % This target is a 10 x 10 pixel rectangle with the top left quarter and % the bottom right quarter white, the rest black target2=[255,255,255,255,255,0,0,0,0,0;255,255,255,255,255,0,0,0,0,0; 255,255,255,255,255,0,0,0,0,0;255,255,255,255,255,0,0,0,0,0; 255,255,255,255,255,0,0,0,0,0;0,0,0,0,0,255,255,255,255,255; 0,0,0,0,0,255,255,255,255,255;0,0,0,0,0,255,255,255,255,255; 0,0,0,0,0,255,255,255,255,255;0,0,0,0,0,255,255,255,255,255]; %create copy of image and insert targets fimcor=fim; fimcor(Row-3:Row+2,Col-10:Col+9)=target; fimcor(Row2-5:Row2+4,Col2-5:Col2+4)=target2; subplot(2,3,4); imshow(fimcor); imwrite(fimcor,'car_corresult.bmp'); % %calculate moment invariants of template tem2=im2double(tem); %invmoments() from DIPUM temp_phi=abs(invmoments(tem2)); % %convert sobel of input to binary for boundary location - use a % threshold of 0.25 (corresponding to 64 for 8-bit) fbin=im2bw(fimsobel,0.25); % %perform erosion, then dilation to try to eliminate protrusions from %the license plate SE=[0,1,0;1,1,1;0,1,0]; fbinopen=imopen(fbin,SE); % %find boundaries of objects in image %boundaries() from DIPUM

19

B=boundaries(fbinopen); % %create image of all boundaries b=cat(1,B{:}); %bound2im() from DIPUM bim=bound2im(b); bims=im2uint8(bim); imwrite(bims,'boundaries.bmp'); subplot(2,3,5); imshow(bims,[]); % %find location of rectangles containing each boundary NumB=size(B); for i=1:NumB %find and store the max and min for x and y of each boundary rcmax(i,:)=max(B{i}); rcmin(i,:)=min(B{i}); %create image containing boundary fsub=fim(rcmin(i,1):rcmax(i,1),rcmin(i,2):rcmax(i,2)); %calculate and store the moment invariants for subimage fsub2=im2double(fsub); fsub_phi(i,:)=abs(invmoments(fsub2)); %compare to template values and store difference fsubdiff(i)=norm(fsub_phi(i,:)-temp_phi); end % %find best match [C,I]=min(fsubdiff); % %create subimage of best match fimloc=fim(rcmin(I,1):rcmax(I,1),rcmin(I,2):rcmax(I,2)); %save and display image imwrite(fimloc,'lic_match.bmp'); subplot(2,3,6); imshow(fimloc,[]);

20