Towards more Agility in Robot Painting through 3D Object Recognition

A. Pichler, H. Bauer, C. Eberst, C. Heindl, J. Minichberger

Robotics and Adaptive Systems, PROFACTOR Research, 4407 Steyr-Gleink, Austria

Abstract
Building to order implies responding to individual customer requests, not simply producing large numbers of goods
for stock and encouraging their sale by promotion and discounting. The paint shop is one of the biggest bottlenecks
in production today. Adapting state-of-the-art robotized painting lines to new variants is time-consuming and leaving
the line in a non-productive state. To overcome the limitations of state-of-the-art robotizes painting system - that are
economically not viable at small lotsizes, the Flexpaint project (2000 - 2002) did develop a new "what you see is
what you paint" approach. This approach scans unknown parts being transported to the painting cabinet, reconstructs
their geometry and painting-relevant features, and automatically plans the painting strokes and executable - collision
free - trajectories. Finally robot code is automatically generated and executed by the painting robots. The Flexpaint
system poses a new paradigm in the frame of agile manufacturing. Resorting to 3D sensing technology in a precedent
step to the task planning process proved the fitness of that concept for small volume high variant painting in experiments.
This paper presents an extension of the "what you see is what you paint" approach by means of dynamic vision and
3D object recognition that is able to complement missing data of scanned parts by use of CAD information of recognized and localized parts.
Keywords: Automatic robot programming, industrial robotics, 3D computer vision.

1. Introduction
Production on demand, mass customisation, rapid
reaction to market changes and quick time-to-market
of new products and variants at small batches are
needed - at low cost and high quality. As investments
in automatic painting lines are considerably high and
as the painting line often is the bottleneck in production, it is imperative to prevent non-productive times
and maximize the use of the expensive equipment.
Aforementioned shrinking volumes and increasing
variances challenge the state of the art. Highly flexible, scalable and user-friendly production equipment is
needed, including robotic systems for painting - a
common process in production. The presented work

responses to the need to overcome current limitations.

This paper presents an extension of the "what you
see is what you paint" approach by means of dynamic
vision and 3D object recognition that is able to complement missing data of scanned parts by use of CAD
information of recognized and localized parts.
The proposed system consists of a selfprogramming robotic cell with embedded simulation
and planning that autonomously generates the robot
programs based on CAD and sensor-data or even pure
sensor data. Thus, it can operate within the digital
factory while at the same time compensating challenges as (1) in-available data, (2) large disturbances
and deviations (shape, position, or even part type) and
(3) even very small lot-sizes. From control perspec-

Fig. 1. System overview of the sensor based

robot painting system

tive, flexibility and robustness of production is to a

large extent defined by (1) needed efforts or speed for
task planning and programming (teach in, OLP), (2) by
its level of communication with respectively its integration into the digital factory, its scheduling capabilities in dynamic environments, and (3) by its autonomy
to react in real-time for initially unforeseen part tolerances and position deviations.
Bottleneck task programming: Modern CADbased off-line programming (OLP) tools partially solve
deficiencies of industrial standard teach-in and iconbased based programming such as the insufficient
speed for programming and/or the (permanent or
partly) blockade of the equipment in an unproductive
state. Automatic planning and programming speeds up
this process further. Started with milling and rapid prototyping applications [15,22,6], research in automatic
path planning is mostly focusing on products consisting of geometrically and technologically ideal parts
and their topological relations. Randomized road maps
or hierarchical best-search (A*-) algorithms [2] reduced complexity and improved planning, and led to
the recent introduction of the first automatic planning
tools. Research in automatic planning is now exceeding the level of linear planning in order to cope well
with disturbances and a high-variant, low-batch production. For parts with very limited surface curvature
and with available full 3D models, the SmartPainter
project follows a novel approach to plan painting motions: (1) virtually folding out the surfaces to be
painted, (2) plan the painting motion, (3) folding back
the surfaces and the painting motions [7,18]. Automatic generation of robot programs for welding and
painting has been presented in [10,14].
Challenge uncertainties: In opposite to high-level
uncertainties (incl. non- stable and non-deterministic
production disturbances such as rush orders) that challenge scheduling algorithms within the (digital) factory, low-level uncertainties can widely be managed
locally at the equipment by incorporation of sensory
information. While vision-based control to compensate

small pose deviations (usually in 2D or 2 1/2D) and

calibration errors are state of the art, large deviations in
object shape or 3D/6DOF pose corrections at complex
objects are beyond. The range image processing to
compensate missing data or pose uncertainties described in this paper includes segmentation, feature
extraction and recognition/localisation. Related work
on segmentation and is presented in [9], on finding
features with defined geometric properties in [3,16].
Recognition and localisation of objects in 3D based on
range images has been described in [20].
Related approaches to compensate large uncertainties using planning based on sensory data have been
presented in the original "FlexPaint" project [1], the
"Fibrescope" Project (Flexible inspection of Bores with
an robotic Endoscope) and in [21], where a sensory
approach is used to deal with uncertainties in turbineblade repair was introduced. There are are numerous
3D object recognition methods that are either global,
like eigenpictures [17] or eigenshapes [4], or that rely
on an initial segmentation of the object [8,5]. Those
methods obtain good results on noise free images, but
there deficiencies on global properties which makes
them vulnerable to occlusions. A more generic way of
approaching the object recognition problem pose spin
images [12], which have been shown to yield
good results with cluttered or occluded objects. As this
method is based on finding correspondences between
image and model regions, it is rather time intensive,
though. [11] gives a good overview about current
global and local approaches on range images.
The first part of the paper describes the Flexpaint
system, its evaluation in industrial manufacturing lines
and its existing limitations at highly complex scenes.
The second part describes the extension of the approach by dynamic vision and object recognition and
localization - and demonstrate its impact on compensating incomplete data and disturbances that are induced by the industrial environment.
2. Sensor based Robot Paining
2.1. System Description
The main concept foresees to derive the paintprocess programming automatically even if the product
is unspecified [19,23]. The proposed robot-system
analyses product sensor data of the goods to be
painted. Process critical regions of work pieces are
identified by tailor-made feature detectors, adopting
human painting knowledge to meet the required quality
of the paint process. Established mappings between a
particular feature of this pre-processed (interpreted and

Fig. 3. Left top: part geometry is acquired by 3D laser

scanner; right top: registrated 3D points cloud; left bottom: reconstructed surface model; right bottom: determination of process specific features.

filtered) data and a process model allow deriving paint

strokes by a process-planning system that is based on
an inverse approach. Feature-specific paint-strokes
for the feature-sets and sequences are converted to collision free tool and robot motions and complemented
with airmotions to a full speed-optimized paintapplication. Automatically generated executable programs are first simulated and than executed on the
paint-robots.
2.2. Geometry Detection
A 3D sensor cell that scans the parts to be
painted during transportation, before entering the
painting-cabinet allows to significantly reduce the constraints on data completeness and positioning of the
parts to be painted. The concept foresees a full integration into the factory as it provides both rapid taskprogramming and task-simulation prior to execution
(commonly) shorter than execution time. The principle
approach to programming the painting robots is an
inverse approach or a "wysiwyp" (what you see is what
you paint) approach. The part to be painted is scanned
while transported to the paint-cell. Range images of the
part scanned during their transport on the conveyor are
taken based on the sheet of light technique. Static
mounted laser-profiling sensors are used and the conveyor motion is employed to provide the feed needed
to reconstruct the third dimension. Moving equipment
is avoided. The cameras are triggered by the encoders.
Wavelength-filters and protection against direct
sunlight raise robustness at surfaces and shapes with
problematic properties. The sensor data of all cameras
are filtered and registered over time to generate a full
3D model of the part or the parts. Data from part(s) and
the hooks need to be separated. The sensor data must
be reduced to a level that is handable for the following
steps and interpreted according to elementary geometry
types that reflect the constraints of the painting process.

Fig. 2. FlexPaint clone exposed on PaintTech 2004 (trade

fair for painting and powder coating technology)

The part surface/shape is to be categorized into a generic surface fraction that can be reached optimally
by the paint-fan with the gun oriented in defined orientation to the surface and the paint-stroke. Cavities and
rips and customer-specific features are recognized in
parallel as they need to be handled differently by the
paint-process and thus the planning tool.
2.3. Collision free paint path generation
Next the idea of the employed paint-process
planning is to link elementary geometries to a process
model. This link is established with a flexibility that
considers that the precise painting strategy that is
mapped to the geometries may vary from customer to
customer. Scheduling of individual strokes follows
specific criteria, as cycle time or others. Next, the sensory retrieved pose of the part are employed
by the AMROSE collision avoidance SW to plan collision-free robot paths the paint trajectories planned by
the INROPA paint-planner. The reconstructed shape of

Fig. 6. Examples of parts hanging in a skid in an industrial setup. The scenario is challenging since the complexity of several parts lead to large occlusion and parts
are oscillating

the part(s) and the hooks and the representation of the

entire work-cell and active components are used to
simulate the collision-free path prior to execution. If
the proper execution can be guaranteed, a generic robot
program is generated.

Finally, the generic program is parsed and converted to the robot-specific program and executed.
2.4. Experimental evaluation Sensor based robot
painting
Sensor data acquisition includes reconstruction,
sensor-data fusion, and extraction of process-relevant
features. Results of the individual steps are visualized
in the following.
As can be seen in the Figure 3, all painting-process
critical features of the gear-box (cavities and other
parts of the object that are hardly reachable) have been
detected despite imperfect sensor data. A remaining
challenge is the handling of regions of the part which
are completely invisible to the sensors because of (i)
occlusion or (ii) surface properties (as at the front of
the gear-box).
Automatic generation of the programs for controlling the robot and the cell is achieved within 60 seconds to 300 seconds, depending on the complexity and
the numbers of the objects and the (density of the)
structure of the environment.

Generally the recognition task is considered as matching task between two surfaces. The proposed 3D object
recognition scheme is based on spin images which do
not impose a parametric representation on the data, so
they are able to represent surfaces of general shape.
Finding correspondences of spin images between
model and scene points. A loss function of the correlation coefficient is used as measure of similarity.
Finding correspondences using the correlation coefficient is computationally expensive, and therefore, a
different way of managing the information conveyed

Fig. 5. Left: simulation of the planning trajectories; right:

execution of paint task

3. Dynamic 3D Object Recognition

As mentioned in the above section following the
approach what you see is what you paint has its limitations to meet paint requirements for highly complex
shaped parts and non separated parts due to laser and
camera shadows. Another issue waiting in the wings
are the goods hanging and oscillating in the conveyor
and their dimension. To counter these issues a main
innovation has been added to the whole robot paint

Fig. 4. Planned tool paths for the painting process

system: a robust 3D object recognition procedure to

retrieve CAD models from fragmented 3D laser scans
which is presented in the paper.
As seen in Figure 8 complete object models are
searched for in a scene. The outcome of the algorithm
provides the correspondence between scene and model.

Fig. 8. System diagram of object recognition approach

by the spin images is needed. In order to make the

process of matching efficient, dimensionality reduction
was achieved by projecting spin images represented as
n-tuples to a space of dimension d<n , using principal
component analysis (PCA). Spin images provide a
comfortable way of handling generic 3D data. Unfortunately in spite of matching PCA compressed 3D data
becomes an extensive task when dealing with hundreds
of 3D models as common in industry. Furthermore
spin images are quite redundant and tend to provide
ambivalent results. These known issues in mind the
algorithm has been extended by a key point detection
algorithm to find the most representative points of a
3D model. These key points are interest or salient
points which can be distinguished from each other.
According to the definition such features have maximum saliency. The term saliency features has already
been used by many other researchers [13].

Finding most salient features in 3D spin image

data has been approached by measuring the information complexity resorting to computing the entropy of
each 3D point. Having detected the key points of 3D
models a database is indexed.
A database contains a set of models. Each model is
represented by vector of n key points. Recognition
consists of finding the model which correspondents to
a given scene that is the model which is most similar to
this scene. Randomly selected points are taken from
the scene and nearest neighbors are evaluated in the
model database. If the distance of a model to a scene
point is below a threshold the corresponding model
gets a vote. The idea of the voting algorithm is to sum
the number of times each model is selected. The model
that is selected most often is considered to be the best
match. In a verification step all selected models are
tested against the scene by matching labeled scene
points to corresponding models using Iterative Closest
Point. The Hausdorff distance between scene and
model points has been used as measure of the quality
of the recognition result.

periments, the frame and the hooks are black and shiny
absorbing most of the laser light. As a result of this
physical effect the frame is more or less shredded.
Hooks have a quite flimsy geometry and barely reveal
a dense point cloud. In our application, the main purpose of the object recognition algorithm is to robustly
label 3D scene point data with the identification numbers of the 3D models kept in a database. In a subsequent step the corresponding models are matched
against labeled scene data to retrieve position and orientation of CAD models related to the world coordinate space which is of importance for the robot application. Figure 10 depicts the recognition result for
each of the test objects. Apparently, the position and
orientation has been estimated subsequently. The robustness of the object recognition algorithm strongly
depends on the quality of the sensor data. Much care
has been taken to generate a smooth surface model
suppressing most of the outliers as surface normals
determine the quality of the basic feature spin image
[12].
4. Conclusion

2.4. Experimental evaluation 3D Object Recognition

The recognition algorithm and the position estimation were tested in combination. A set of 25 objects
hanging in conveyor frame was selected. It basically
contains 4 different industrial objects hanged up in
various positions and orientations. While the conveyor
was moving around 55 scans have been taken by the
sensor dynamically adapting its own kinematic configuration to cover the entire frame. Taken scans are
registered and transformed to a surface model (Figure
9). Apparently, the objects in the frame causing a vast
amount of occlusions due to their proximity. A next
issue which is often encountered in industry is the
loose hanging of objects in the frame resulting in tumbling objects. Different colors of the objects to be
scanned heavenly affect the outcome of a 3D laser sensor system. Particularly, in the test set used for the ex-

The "what you see is what you paint" approach has

proven to be a promising alternative to conventional
teach in and OLP based programming and has shown
feasible especially for high variant, low volume parts.
However, highly complex shaped, non separated and
oscillating parts cause incomplete data (by occlusion)
and disturbances. As pointed out incomplete surface
models hinder the paint planner to generate paint
strokes resulting in a more predicable paint coating
thickness. Furthermore, collision free motion planning
requires some safety distance from incomplete objects
preventing the system to paint more concave shaped
objects. Robust 3D object recognition and pose estimation contributes to the realization of small lotsize robotic painting applications if part or scene complexity
is very high.
Future work will focus on increasing the robustness of the recognition algorithm against cluttered sensor data.
Acknowledgements

Fig. 9. Reconstructed parts hanging in a skid

The authors would like to thank the European

Commission and the partners of the Innovative Production Machines and Systems (I*PROMS) Network of
Excellence for their support under the Sixth Framework Programme (Contract No. 500273). PROFACTOR is core member of the I*PROMS consortium.
Further information is available at: www.iproms.org

References
[1] Flexpaint. [Online]. Available: www.flexpaint.org
[2] Autere, Resource allocation between path planning algorithms using meta a*, in ISRA, 1998.
[3] N. W. C. Robertson, R.B. Fisher and A. Ashbrook, Finding machined artifacts in complex range data surfaces, in
Proc. ACDM2000, 2000.
[4] R. J. Campbell and P. J. Flynn, Eigenshapes for 3D object recognition in range data, pp. 505510. [Online]. Available: citeseer.ist.psu.edu/137290.html
[5] O. Camps, C. Huang, and T. Kanungo, Hierarchical
organization of appearance-based parts and relations for object recognition, 1998. [Online]. Available: citeseer.ist.psu.edu/camps98hierarchical.html
[6] E.Freund, D. Rokossa, and J. Rossmann, Processoriented approach to an efficient off-line programming of
industrial robots, in IECON 98: Proceedings of the 24th
Annual Conference of the IEEE Industrial Electronics Society, 1998.
[7] P. Hertling, L. Hog, L. Larsen, J. Perram, and H. Petersen, Task curve planning for painting robots - part i: Process
modeling and calibration, IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 12, no. 5, pp. 324
330, April 1996.
[8] R. Hoffman and A. K. Jain, Segmentation and classification of range images, IEEE Trans. Pattern Anal. Mach. Intell., vol. 9, no. 5, pp. 608620, 1987.
[9] A. Hoover, G. Jean-Baptiste, X. Jiang, P. J. Flynn, H.
Bunke, D. B. Goldgof, K. K. Bowyer, D. W. Eggert, A. W.
Fitzgibbon, and R. B. Fisher, An experimental comparison
of range image segmentation algorithms, IEEE Transactions
on Pattern Analysis and Machine Intelligence, vol. 18, no. 7,
pp. 673689, 1996. [Online]. Available: citeseer.csail.mit.edu/hoover96experimental.html
[10] N. Jacobsen, K. Ahrentsen, R. Larsen, and L. Overgaard, Automatic robot welding in complex ship structures,
in 9th Int. Conf. on ComputerApplication in Shipbuilding,
1997, pp. 410430.
[11] R. J.Campbell and P. J. Flynn, A survey of free-form
object representation and recognition techniques, Comput.
Vis. Image Underst., vol. 81, no. 2, pp. 166210, 2001.
[12] A. Johnson and M. Hebert, Using spin images for efficient object recognition in cluttered 3d scenes, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.
21, no. 5, pp. 433 449, May 1999.
[13] T. Kadir and M. Brady, Scale, saliency and image description, International Journal of Computer Vision, vol. 45,
no. 2, pp. 83105, 2001.
[14] K.K.Gupta and A. D. Pobil, Apartical motion planning
in robotics: Current approaches and future directions, 1998.
[15] K. Kwok, C. Louks, and B. Driessen, Rapid 3-d digitizing and tool path generation for complex shapes, in IEEE
International Conference on Robotics and Automation, 1998,
pp. 27892794.
[16] D. Marshall, G. Lukacs, and R. Martin, Robust segmentation of primitives from range data in the presence of

Fig. 10. Visualization of recognition result: Reconstructed sensor data (green shaded) and recognized
matched CAD model (grey wire frame).
geometric degeneracy, IEEE Trans. Pattern Analysis and
Machine Intelligence, vol. 23, no. 3, pp. 304314, 2001.
[17] H. Murase and S. K. Nayar, Visual learning and recognition of 3-d objects from appearance, Int. J. Comput. Vision, vol. 14, no. 1, pp. 524, 1995.
[18] M. Olsen and H. Petersen, A new method for estimating parameters of a dynamic robot model, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17,
no. 1, pp. 95100, 2001.
[19] A. Pichler, M. Vincze, O. M. H. Anderson, and K.
Haeusler, A method for automatic spray painting of unknown parts, in In IEEE Intl. Conf. on Robotics and Automation, 2002.
[20] F. R.B., F. A.W., M. Waite, O. M., and E. Trucco,
Recognition of complex 3-d objects from range data, in
CIAP93, 1993, pp. 509606.
[21] X. Sheng and M. Krmker, Surface reconstruction and
extrapolation from multiple range images for automatic turbine blades repair, in IEEE IECON Conference, 1998, pp.
13151320.
[22] W. Tse and Y. Chen, A robotic system for rapid prototyping, in IEEE International Conference on Robotics and
Automation, 1997, pp. 1815-1820.
[23] Bauer, A., Eberst, C., Nhmeyer, H., Minichberger, J.,
Pichler, A., Umgeher, G. , Self-programming Robotized
Cells for Flexible Paint-Jobs, International Conference on
Mechatronics and Robotics 2004, Aachen, Germany.