Automated Inspection of Flexible Assembly: System Design and
Algorithm Research
Peihan WEN, Li ZHENG, Jian ZHOU, and Burton CLARKE
Abstract-Quality control plays an important and key role in
manufacturing process. This paper defines a problem realizing
real-time quality control based on image processing and sensor
network, divides the problem into four sub-problems, as well as
puts forwards the overall schema to solve the problem. Then the
first sub-problem, Object Recognition, is focused on, its system
design pattern is presented, and main algorithms of each step
are studied. Some experiment results are given to show the good
performance of the proposed system for object recognition
sub-problem.
I. INTRODUCTION
Q UALITY control plays an important and key role in
manufacturing process. With the development of
computer science and machine vision, automated on-line
quality inspection and control is more and more widely
applied in industry. Wu et al. [1] introduced the development
of an automated visual inspection system for printed circuit
boards (PCBs). It utilized an elimination-subtraction method
which directly subtract the template image from the inspected
image, and then conduct an elimination procedure to locate
defects in the PCB. Each detected defect is subsequently
classified into one of the seven defect types by three indices.
Kosmopoulos et al. [2] introduced a system for automated
gap inspection that employs computer vision. It is capable of
measuring the lateral and the range dimension of the gap
(width and flush, correspondingly). Lahajnar et al [3]
presented a machine vision system for automated visual
inspection of plates of electric cookers, with the goal to
reduce~~~~~~~~~~~
lao.otadesr ossetpoutqaiy
reduce labor cost and ensure consistent product quality.
Derganc et al. [4] described a machine vision system for
inspecting bearings, which are an important part of
electro-mechanical kWh meters. Antonelli et al. [5] presented
the design and development of an architecture that is able to
provide remote segmentation service to various kinds of
images or applications. The system exploits the
functionalities offered by the already existing desktop
application Isocontour by extending its capabilities toward a
client-server environment.
This work was supported in part by Caterpillar Inc.
P. WEN is with the Department of Industrial Engineering, Tsinghua
University, Beijing, 100084, China (e-mail: WenWphAgmail.com).
L. ZUENG is with the Department of Industrial Engineering, Tsinghua
University, Beijing, 100084, China (corresponding author to provide phone:
86-10-62785584; fax: 86-10-62794399; e-mail: lzheng@tsinghua.edu.cn).
J. ZHOU is with the Department of Industrial Engineering, Tsinghua
University, Beijing, 100084, China (e-mail: jian@tsinghua.edu.cn).
B. CLARKE is with Advanced Operations Technology & Process Quality
Technology, Caterpillar Inc., Peoria, IL., 61630-0299, USA (e-mail:
clarke_burton_r@cat. com).
1-4244-2368-2/08/$20.OO ©2008 IEEE
However, there are few papers on flexible assembly where
the assembly object are flexible and there are different kinds
of assembly shapes, e.g., the hydraulic hose between two
joints. Since the assembly shapes are not definite, some
traditional inspection methods applied in rigid assembly are
not appropriate. H.S. Ip. et al. [6] treated the flexible objects
such as blood vessels as two parallel curves and recognized
the object by detecting the parallelism. The problem of
parallelism detection between two curves was formulated as a
line detection problem within an affine-invariant local
similarity matrix computed for the two curves. Each element
of this matrix gives an affine invariant measure of local
parallelism for any pair of curve segments along the two
curves. This method is a little complicated in modeling and
sometimes not proper under some conditions. So the method
of image segmentation comes into our sight and shows
benefits on complexity and application.
In this paper, we define the real-time quality control
problem based on image processing and sensor network,
divide it into four sub-problems, and focus on object
recognition sub-problem, present the system design pattern
and algorithms of some key steps. The rest of this paper is
organized as follows. Section II gives the problem definition
and sub-problems classification. Section III sketches out the
overall schema of the problem. Section IV focuses on the
object recognition sub-problem, gives the system design
patter, and presents some key algorithms. Section V shows
some experiment results and Section VI draws the conclusion.
s
A. Problem Description
Quality is an important attribute of products, and also life
of manufacturing factories. Nowadays, the most used quality
ensuring methods are quality inspection and quality control.
Quality inspection is to inspect the products after each stage
of and the whole process to ensure there are no defects,
whereas quality control is to apply some rules obtained by
theory or empirical data in manufacturing process to avoid
possible defects.
In real industrial environment, we want the time of quality
inspection to be as short as possible and the accuracy of
quality inspection to be as high and consistent as possible.
And also we want to record the inspection results and some
characteristic parameters automatically during inspection
process to do further analysis to help improving quality
control methods.
222
 Object Extracdtioni

- Edge Pointg Extrcion

Edge Points Fitting

Fig. 3. Object recognition system module flow chart.

B. Problem Decomposition
The above proposed problem contains two stages: 1)
on-line quality inspection and control, 2) off-line data
analysis. Before these processes, all the assembly units
Fig. 1. Example flexible assembly image, where the black hose is the (objects) need to be recognized. Therefore, the proposed
focused object. problem can be divided into four sub-problems: object
These real industrial requirements call for computer aided recognition, quality inspection, data analysis and quality
automated inspection (CAAI). There have already been some control.
kinds of systems to perform CAAI for rigid assembly, thus The first sub-problem object recognition is to recognize all
our problem is to develop a system and corresponding main objects from the input images and record the key
algorithms to perform CAAI for flexible assemblies, e.g., as characteristic parameters; the second sub-problem quality
shown in Fig. 1, and real-time record necessary characteristic inspection is to compare key characteristic parameters of
parameters to do further statistic analysis and then give input real-time images with the standard image, inspect
quality control advice. whether the assembly expressed by the input image is correct,
and record the result as well as main comparison data; the
third sub-problem data analysis is to analyze all the recorded
Obj_ct
Recognition Input_Image data during object recognition and quality inspection, find
some statistic rules to improve quality control methods as
E g y $ 1 * Stndiardmage
object o fZ i mn image& well as quality inspection accuracy and efficiency; the final
h> ky c a -ractc
Riord rL step quality control is to apply all the real-time quality
guaranteeing methods to ensure the quality of products
On-line according to data analysis outcome.

III. OVERALL SCHEMA

Kpr= arepara
EComFfwp}With
nuak
ii
#am6Atb
th
-: 6f
MithdAid
ri ~P 141 ,
( hispqect6dri
fth4kd 1
rFig. 2 shows the overall solution schema of the proposed
RLC Jip&tiLbtio r.ftiltuAin lIhk problem. Images are taken by several cameras surrounding
llthrrf
ii eporidir ra eit the assembly workbench. Considering each camera as a
O_ine sensor node, these cameras form a sensor network to give a
Da Analysis whole description of the assembly scene.
The standard images are taken by the sensor network
iS1 Addlyi the id&O.bd di'td to
ol-line get statistic rules, cameras from one definitely correct assembly; they are the
* iV, advICe io improv same for an assembly pattern. The standard images are used
quialilyir3Pcl1 m1tlro
whfibJ to obtain standard characteristic parameters for quality
33chod. l 1inspection. The inspected images are taken real-time, they are
2 ,, ,=. used to express current assembly state and be compared with
Qality Contrl 7
the standard images on characteristic parameters to judge
* Apply ~i1Ilie
renl-iiiiic ~ whether the assembly state is correct.
| u rrecing mehods oensure Before the inspection process starts, several (usually about
~~hcproducls.
qualityof ten) standard images are taken by the camera sensor network;
these standard images are inputted to the inspection system to
Fig. 2. Overall schema: inspection process process and obtain key characteristic parameters; the average

223
 (0,-) (0,1)
Fig. 4. N8 neighborhood
of these parameters will be set as the standard parameters of
this kind of assembly, and used for corresponding inspection.
Then the inspection process is started, and performs
real-time image capturing, on-line quality inspection, and
real-time quality control. The inspection results are real-time
reported and shown on screen, which helps workers to ensure
assembly quality and put right assembly faulty in time. After
assembly process ends, off-line data analysis is performed
with all the data recorded during process running; statistic
rules and corresponding quality inspection and control advice
later inspection process.
will be concluded forrlater
Thllbe ovcluderall inspection process.doesnot
onlyguaImage preprocessing
assem querallinspectyon-line,sbut
assembly quand improves qualityasorecds uulydrantae
inspection
asond corfr remove
strteine contlysisnuouimproves quality inspection and control
IV. OBJECT RECOGNITION
Theiprped problem contains four sub-problems object
T pl and in
this paper, wearefocused on the first sub-problem, object
recognition. Since there are lots of researches on rigid object
recognition, here flexible objects are our points of interest,
e.g., black hoses in Fig. 1.
A. System Design
The object recognition system can be divided into five
main steps, image preprocessing, image segmentation, edge
points' extraction, edge points fitting and characteristics
judgment, as illustrated in Fig. 3.
Since nearly all images subjects to various noises resulting
from the uneven illumination condition, dirt, and so on, image
preprocessing is required, consisting of two steps, noise
reduction and image enhancement, as detailed described in
Part B of this section.
After preprocessing, the object region should be extracted
from the image at the stage of object extraction, which,
however, as detailed reviewed later in Part C of this section,
merely roughly extract the object region.
In the object region, using some edge detection technique,
e.g., Canny edge detection, the contour of the object can be
found. The constraints of edge points presented in Part D of
this section assist to extract all the edge points and classify
them into different groups corresponding to different edges.
Curve fitting is performed with the extracted and classified
edge points on each edge; an approximate model is
A'
dg= d1./cosa
Fig. 5. Distance between corresponding points on both edges.
constructed to describe each edge, as detailed introduced in
Part E of this section.
In the final step, according to the description models of all
the edges, the characteristic parameters of the object are
computed, and then they are used to judge whether the
extracted object is the expected object, if yes, object
recognition is accomplished; if no, there is no the expected
object, as detailed described in Part F of this section.
B. Image Preprocessing
is important and necessary step of
an
image segmentation
and pattern recognition. It
a series of random effectively
can
noise, and enhance the parts of
interest. There two kinds of image preprocessing methods:
are
space domain techniques and frequency domain techniques.
We usually use linear or nonlinear smoothing filtering (in
space domain) and low pass filtering (in frequency domain) to
eliminate or weaken noise effect, whereas we use linear or
nonlinear sharpening filtering (in space domain) and high
pass filtering (in frequency domain) to enhance the specified
regions.
Besides, region growing is also used to help improving the
result of image preprocessing.
C. Object Extraction
In this paper, we do image segmentation to extract the
object region from the original image. There are a good many
image segmentation techniques, which can be classified into
four categories: feature domain based techniques, space
domain based techniques, hybrid techniques, and special
theory based techniques. The feature domain based
techniques only consider feature information of the image,
the space domain based techniques only take the space
information of the image points into consideration, whereas,
hybrid techniques consider both feature and space
information. Special theory based techniques apply some
special theories such as fuzzy theory, neural network, wavelet
analysis, etc., to do image segmentation.
We proposed an effective hybrid algorithm, spatial
credibilistic clustering algorithm (SCCA), using both feature
and space information, and introducing credibility measure
into image segmentation in [7].
Firsty e1etefloignttos
stydeieheflongoain:
224
 of the center pixel Xh,w to its neighbor Xh+lh,w+lw. The larger
Ah,w the more independent the center pixel.
1
Isolateathe Points f teetWo +1 e-(-K)/U (4)
1b undaties
where the parameters K and ca specify the displacement of A
No from zero and the steepness of A respectively. A reasonable
Is the real curve X-Y style estimate for K and a- is
nh nw
Yes Transin
.I I1 rlt
1
K = NAA11avv(Xh,w), (5)
Remove the reated poinfts
t
qyle h=1 w=1
0.95Qav- K
IRemoe
th eoipits e h,w) 1n4 (
+ ~~~~~~~~where
Sinothth&a
6 1
+ (
av (Xh,w) = (h,w),(h+h,W+w) (7)
Fit eakh edge points lhh=)-1+l=+
The above dav and K can be regarded as the average
distance of the center pixel to all its neighboring pixels and
the average distance of all the pixels in the whole image to its
Fig. 6. Flow chart of object edge points fitting. neighboring pixels respectively, in N8 neighborhood.
The objective function of SCCA is
G ={Xh,w h = 1,2, ...,nh, w = 1,2,_...,nW: an nhX nw c nh nw
image, where h,w are coordinates of each pixel in the JSCCA(Cr, A) = (Cri,h,W)mD,h,W (8)
image and Xhw (vector) represents the feature of the pixel i=1 h=1 w=1
(h, w); define N = nh x nw for convenience; subject to
- c: a positive integer with c > 1 representing the number of max c{f(Dk,h,w)1,
1i,h,w = f(D,h,w)/ 1<k<c (9)
clusters; where
- 0 = {fl, 02 ...J, Ocl: set of clusters with 0i representing the - Cri,h,w = Cri (Xhw) a positive number representing the
ith cluster, i E I = {1,2, ... C}; I credibility measure of feature points Xhw belonging to
- A =
(a,, a2, ..., ac): cluster center matrix, where ai (vector) fuzzy cluster Oi,
represents the feature ofthe ith cluster center for each i E I; 1 up
dL,h,w: a distance from a pixel Xh,w in the image to a cluster Cri,h,w = 2 i,h,w + 1 -SUp k,h,w); (10)
center ai in the feature space with - Cr = {Cri,h,w i E I, (h, w) E GI: a c x N matrix denoting
di,h,w = ||Xh,w - ail; (1) the credibility measure matrix;
Then, in N8 neighborhood shown in Fig. 4, where the - Mi,hhw: a positive number representing the possibility
center pixel is marked by a cross. Let Xh,w denote the center measure of feature points Xh,w belonging to fuzzy cluster
pixel, then the neighboring pixels can be denote as Xh+lhw+1w, 0i;
with Ih, lw E {-1,0,1} and (lh, lw) . (0,0). - f: a function of Di,h,w, satisfying the following equation,
The dissimilarity of the center pixel Xh,w and its neighbor 1
Xh+lh,w+1w is measured using the following Euclidean f(DL,h,w) In 99' (11)
distance l+ iDh, w
0 )(h,w),(h+lh,w+Iw)
= Xhw Xh+lh,w+lwI (2) 40
If the distance 0(h,w),(h+lh,w+Iw) is small (i.e., Xhw and - m > 1: the fuzzifier which has a great influence on the
Xh+lh,w+lw are similar), di,h,W should be greatly influenced by performance of fuzzy clustering.
di,h+lh,w+lw, otherwise, di,h,W should be largely independent
of di,h+lh,w+lw. Taking all pixels in N8 neighborhood, the The update equation of A is
dissimilarity index D1hW which measures the dissimilarity nh l ()nwl h h,w
between Xh,w and the ith cluster center ai can be given as i nh Enwfor ECIi (12)
follows: with
1
D 8hW = 1 l -- Xh,w =
1 wi1Y 1

+dth+lhw+lw(1 - 2lw)1, (1h lw) .

(0,0) (3) ; xh+lhw+W(1 21h$tw)1 (1h}
-
w) (0,0). (13)

where it,W= i(d(h,w>,(+111,w+1w)) = At(8), o . .t 1 is the D. Object Edge Constraints

independence factor controlling the degree of independence In the step of object extraction, the feature and space

225

(a) (b) (c)
(d) (e)
Fig. 7. Experiment result of hose 1.
information of the image is utilized, whereas in this step, we
want to incorporate the shape information of the object to
refine the extraction result and pick up edge points more
accurately.
Each object has a variant or invariant shape. Even a flexible
object whose shape is not definite, there must be some
characteristic shape information that is invariant. Take the
hoses in Fig. 1 for example. The shape ofhoses is variant, and
very hard to model, but each section ofthe hose is a circle and
the diameter is a constant if there is no radial external force.
So we can choose the diameter as a characteristic shape
parameter to form some constraints to the edge points.
Let dr be the diameter of each section circle, i.e., hose
diameter, di be the vertical distance between two
corresponding points A and A' on both edges, and ai be the
angle between radial direction and vertical direction as shown
in Fig. 5; we have the following relationship equation
di dr
=
cosa,
If the hose is horizontal, ai E [0, wT/4], then di E [dri 1.4dr];
if the hose is vertical, just rotate the axis and the situation is
similar, still di E [dri 1.4dr].
Then the edge point constraint is that the distance of two
corresponding points on both edges must be on the interval of
di . To extract edge points, firstly perform canny edge
detection to find all edge points, and then compute the
distance of the corresponding points on both edges, judge
whether it is on the interval to remove noise points and refine
extraction result.
X ;H~~~~~~~~~two
|X|
j ! : t ,f w ~~~~~~judgment step is indeed parallelism judgment. With the
(a) (br) (C) (d) (e)
Fig. 8. Experiment result of hose 2. . .....
(a) (b) (c) (d) (e)
Fig. 9. Experiment result of hose 3.
E. Object Edge Fitting
E To recognize the object, curve fitting with the extracted
edge points is needed. Fit each edge ofthe object with a curve
and then we can judge whether the extracted object is the
expected object using the characteristics comparison. Again
take the hoses in Fig. 1 for example. The key characteristic of
hoses is that the two edges are approximately parallel. So if
the two fitting result curves are approximately parallel, we
can say the recognized object is a hose and accomplish object
recognition.
The flow chart of edge points fitting is shown in Fig. 6.
After isolating points on two edges, judge whether the hose is
horizontal (X-Y style) or vertical (Y-X style), if it is vertical,
rotate X, Y axis to transform it to X-Y style because all curve
functions are under the condition of X-Y style. Then remove
noise points using the method so-called median error
theorem. Finally, smooth the data points using the
well-known least square principle and fit points on each edge
using cubic polynomial curve fitting.
Median error theorem for removing noise points can be
executed as follows:
1) select one edge as the current edge;
2) for each point on the current edge,
a) calculate the distances between it and all points on
14
the other edge,
(1) b) pick up the smallest distance as the distance of the
point to the other edge, called point-edge-distance;
3) find the median of these point-edge-distances;
4) calculate errors of all point-edge-distances to the
median, if the error is more than one specified threshold,
the corresponding point is classified as noise point and
removed;
5) select the other edge as the current edge, and perform
step 2) - 4);
6) get the noise removing result.
F. Parallelism Judgment
The final step of object recognition is characteristic
judgment. In this step, we'll judge whether the characteristics
of the object is satisfied in the extracted points' set; if yes, we
can say the extracted object is the expected one and finish the
recognition process; if no, we can say the extracted object is
not the expected one and finish the recognition process.
As to the hoses in Fig. 1, the key characteristic is that the
edges are approximately parallel, so the characteristic
results of edge points fitting, i.e., the two cubic polynomial
curves, we have the process of parallelism judgment as
follows:
226
 find the re-drawn hoses are almost the same as the original
hoses, which is consistent with the recognition results and
proves the effectiveness and good performance of the
proposed system pattern and algorithms.

VI. CONCLUSION
This paper defines a real-time quality inspection and
(a) (b) (c) (d) (e) control problem based on image processing and sensor
network, and divides it into four sub-problems: object
Fig. 10. Experiment result ofhose 4. recognition, quality inspection, data analysis, and quality
control. The overall schema to solve the problem is put
1) defieaBoonelcurean t;ecv forward. Then the first sub-problem Object Recognition is
2) onec
randomlyselect s
3)
cur ent curve;
the focused on, and its system design pattern is presented. The
randomly select 100 points on te afive key steps of the object recognition system, and their main
4) calculate the distances of the selected points to the other
curve; algorithms, are studied. The system pattern will be effective
5) cv .e
Judge whether all the distances are approximately equal th this kind of flexible assembly,
for allloihsmyb whereas the details of
ifretfrdfeetknso
wihi th gie. fotrsod
within th
the algorithms may
givenerrorthresholassembly patterns or
be different
objects. The
for
median
different kinds of
error theorem for
if yes, assign the Boolean variant TRUE, and noise removing and the algorithm of parallelism judgment are
if no,ifassign the Boolean variant
n, asigntheBoolan FALSE;
ariat FLSE
very effective on hhoses, pipes, and objects of this kind.
6) select the other curve as the current curve, and perform very
step 3) -5);
effetiv on. es indojof th is kind.
This paper iS only a beginning of the automated inspection
value of tof flexible assembly problem: the problem is proposed and
7) judge the the first sub-problem is solved, the latter three sub-problems
if TRUE, the object is a hose, and
if FALSE, the object is not a hose. will be solved based on the first sub-problem and studied in
later papers.
Note that this section takes the image shown in Fig. 1 for
example to explain the system pattern and algorithms of
object recognition. For other images or objects, the system [1] W. Wu,boards
M. J. Wang, and C.-M. Lin, "Automated inspection of printed
pattern will be the same, but the details of algorithms may be
circuit through machine vision," Computers
in Industry, pp.
103-111, 1996.
different. [2] D. Kosmopoulos and T. Varvarigou, "Automated inspection of gaps on
the automobile production line through stereo vision and specular
reflection," Computers in Industry, pp. 49-63, 2001.
V. EXPERIMENT RESULTS [3] F. Lahajnar and R. Bernard, "Machine vision system for inspecting
Take four hoses in Fig. 1 as examples, some recognition electric plates," Computers in Industry, pp. 113-122, 2002.
[4] J. Derganc, B. Likar, and F. Pernus', "A machine vision system for
results as shown in Fig. 7 to Fig. 10 are given to show the measuring the eccentricity of bearings," Computers in Industry, pp.
performance of the proposed system and algorithms. In each 103-111,2003.
figure, (a) is the original image, (b) is the image after [5] M. Antonelli, S. G. Dellepiane, and M. Goccia, "Design and
(c)istheimageaftrregiongrowing
clustering segmentation, .c
clustering segmentation,
ig implementation of web-based systems for image segmentation and
CBIR," IEEE Trans. Instrumentation and Measurement, vol. 55, No. 6,
and other pretreatments, (d) is the edge image after edge pp. 1869-1877, Dec. 2006.
detection, and (e) is the recognized hose image. [6] D. Shen, H. H.S. Ip, and E. K. Teoh, "Affine invariant detection of
The recognized hose images are re-drawn using the curve perceptually parallel 3D planar curves," Pattern Recognition, vol. 33,
pp. 1909-1918, 2000.
fitting results. Checking sub-figure (e) in Fig. 7 to Fig. 10, the [7] P. Wen, L. Zheng, and J. Zhou, "Spatial Credibilistic Clustering
two edges of the black region are approximately parallel, Algorithm in Noise Image Segmentation," Proceedings of IEEE
the recognized objects are hoses. Then comparing International Conference on Industrial Engineering & Engineering
appafigurently (e)with
sub-figure sub-figure (a)
(e) with sub-figure in Fig. 77 to
(a) in 10, we'll
to Fig. I we'll AIManagement 2007, Singapore, pp. 543-547, Dec. 2-5, 2007

227