Robotic Arm Control in Space with Color
Recognition Using a Raspberry PI
Roland Szabó
Applied Electronics Department
Fac. of Electronics and Telecom., “Politehnica” Un.
Timişoara, România
roland.szabo@etc.upt.ro
Abstract—In this paper a robotic arm control, with color
recognition, implemented on a Raspberry PI, will be presented.
The Raspberry PI is a small super computer which is suitable for
almost any embedded project. To the Raspberry PI is connected
a Logitech C270 web camera or a TP-LINK TL-SC3230 IP
camera and a USB to serial dongle which does the
communication task. The camera pair films the robotic arm and
the USB to serial dongle controls the robotic arm. The color
recognition is done with OpenCV installed on the Raspberry PI.
The robotic arm has glued colored bottle stoppers on the joints
which are recognized with color filtering. These joints are united
with lines and other lines are drawn to be guide lines to some
mathematical computations. Based on these lines a skeleton of the
robotic arm is created, which is an info for the Raspberry PI
about the position in space of the robotic arm. With further
computations the robotic arm can be moved in the desired
position.
Keywords—embedded; IP camera; Raspberry PI; robotic arm;
USB to serial; SCPI commands; web camera.
I. INTRODUCTION
This paper presents a robotic arm control system made
with a Raspberry PI. The Raspberry PI, the card size super
computer, which is the main component of this control system
[1].
The industry needs these kinds of systems which can
control a robotic arm in a smart way and also they are made
around an embedded system [2].
The industry is changing, the German factories made a
new standard called “Industrie 4.0”, where all the devices are
connected with each other via a wireless network. Actually the
robots are not anymore controlled by PCs, but with smart
phones, tablets or other embedded devices [3].
These is a prediction about the future that there won’t be
PCs in the industry, as they are today, there will be only smart
embedded devices which are connected with each other via a
big wireless network [4].
The other characteristic of the robots of the future is that
they need to be smarter and more autonomous. This new
environment cannot be created by having only preprogrammed
robots, which follow a preset path; they have no decision tasks
or vision systems integrated. In the old industrial environment
The authors would like to thank Faculty of Electronics and
Telecommunications, Politehnica University Timișoara for the given support.
978-1-5090-1288-6/16/$31.00 ©2016 IEEE
Aurel Gontean, Alexandru Sfiraț
Applied Electronics Department
Fac. of Electronics and Telecom., “Politehnica” Un.
Timişoara, România
aurel.gontean@upt.ro
the robots needed to be preprogrammed and recalibrated
periodically, which could be done only by stopping the
production line, which meant losing a lot of time and money
[15].
In the new and smart industrial environment the robotic
arms does not require to be preprogrammed, because they can
compute the movement path with the help of their integrated
vision system [16]. More than this, they can auto calibrate
using the images acquired by the cameras, so there is no need
to stop the production line. This was the productivity can be
increased, because in today’s consumer society is a huge
demand of products and this will only grow in the future [17].
In this experiment a smart robotic arm controlling system
will be presented. The robotic arm system is controlled by a
Raspberry PI, which is a compact card size, smart, embedded
system. To the Raspberry PI cameras like a Logitech C270
web camera or a TP-LINK TL-SC3230 IP camera are
connected, these acquire live images, which are then
processed and the robotic arm is controlled accordingly. The
robotic arm has no preprogrammed functions, everything is
decided in real-time. The robotic arm is controlled via a serial
interface by a USB to serial dongle [8].
II. PROBLEM FORMULATION
In the university’s laboratory there was a Raspberry PI
development board, one Lynxmotion AL5A robotic arm, one
Lynxmotion AL5B robotic arm, two Logitech C270 web
cameras and two TP-LINK TL-SC3230 IP cameras. The task
was to put this all together and do a smart robot control
system. For serial communication the Raspberry PI had also
some ports, but there is needed a level shifter, this way the
microcontroller won’t be fried. For serial communication is
needed ±12 V, but the Raspberry PI uses only 0 – 3.3 V,
because of this, the signal’s amplitude is needed to be
increased; level shifter has to be used. Sometimes is easier to
use a USB to serial dongle, because this way there is no need
to complicate the system with other circuitry [9].
III. PROBLEM IMPLEMENTATION
A. Used Equipment
The selected robotic arm was the Lynxmotion AL5B
robotic arm, because is slightly bigger than the Lynxmotion
689 TSP 2016
AL5A robotic arm. Its gears are made from carbon reinforced
metal and not form plastic [10].
The control circuitry of the Lynxmotion AL5 type robotic
arms is the SSC-32 servo control board, which can control up
to 32 servomotors. For the Lynxmotion AL5 type robotic arms
only 6 motors are needs to be controlled, so the SSC-32 servo
control board is more than suitable to do the task. For a 3D
movement even the control of 3 motors is enough. The SSC-
32 servo control board can communicate with a master device,
in the present case with a Raspberry PI, via RS-232 serial port.
The SSC-32 servo control board generates PWM signals on
the digital ports with a specific duty cycle coded in the SCPI
commands and this is sent over the serial interface. The PWM
signal will move the connected motor proportionally with the
preset duty cycle [11].
The robotic arm has special Hitec motors, they have gears
for increasing torque and they have also a small control
circuitry which has 3 signals, 5 V, ground and PWM. At the
PWM input a signal is received from the SSC-32 servo control
board, this way the circuit board will move the motor as much
as the duty cycle of the PWM signal is set. The motors are
blocked mechanically to be able to move only 180°, so they
cannot rotate 360°; they just make very small movements [12],
[13].
The used Raspberry PI is the model B, which is not the
latest model, but for the robotic arm control experiment it
worked just fine.
B. Theoretical Background
Before developing any control application for these robotic
arms, the SCPI commands, for controlling them, need to be
known and understood very well.
On Fig. 1 is presented the block diagram of the robotic arm
control experiment with a Raspberry PI.
The main part of the experiment is the Raspberry PI, which
is like a small computer it has a display output on the HDMI
interface and a mouse and keyboard connected to the USB
port. With a D-Link H4 externally powered 4 port USB HUB
the Raspberry PI model B can have a total of 5 USB ports, 1
Raspberry PI Mouse
model B
USB
USB
D-Link H4
USB HUB
HDMI
USB
Display
Keyboard
Fig. 1. Block diagram of the robotic arm control experiment with Raspberry PI
mouse, 1 keyboard, 1 USB to serial dongle and 2 for the
Logitech C270 webcams. Instead of the two webcams two TP-
LINK TL-SC3230 IP cameras can be also connected to the
Ethernet port of the Raspberry PI with the help of an Ethernet
switch.
One interesting fact about the setup is the usage of the
RS232 Pmod, which actually replaces the serial interface of
the SSC-32 servo control board, because this serial interface
worked better. In other words the SSC-32 servo control board
has a lack of design, because the serial driver chip needs
bigger voltage levels than an embedded microprocessor board
can supply, this way the SSC-32 servo control board can be
controlled only by a PC with a very strong power supply. It
cannot be controlled by embedded microprocessor boards,
which cannot supply high signal amplitudes. The robotic arm
behaves the following way: it wakes up and after it collapses,
because the microcontroller from the SSC-32 servo control
board resets, not being supplied with high signal amplitude.
The solution is quite simple, is the usage of the RS232 Pmod
from Digilent, which actually has a better serial driver chip,
which works on lower signal amplitudes. The RS232 Pmod
can be easily connected to the SSC-32 servo control board;
because this has Rx and Tx signal pins and this way the serial
microchip from the SSC-32 servo control board can be
disconnected with a jumper.
C. Software Implementation
The software was implemented in Python scripting
language using the OpenCV libraries. The software is running
on the Raspberry PI on Raspbian (Debian) Linux “Wheezy”
v7.0 operating system.
The image acquisition is done using Logitech C270
webcams over USB interface or TP-LINK TL-SC3230 IP
cameras over Ethernet interface.
With IP camera, the acquisitioning system is a little bit
different. The IP camera takes pictures which are sent over
Ethernet to the Raspberry PI. If the acquisitioning is done with
a big frame rate, the pictures loaded look like a live stream,
but actually there are more pictures loaded like the frames of a
movie.
Logitech Lynxmotion AL5A
C270 robotic arm
webcams
USB
USB
Peripheral
USB
Digitus USB to RS232
Pmod
serial dongle RS-232
690
 Unfortunately on Ubuntu Linux 12.04 LTS installed on PC
works fine the OpenCV function for acquisitioning images
over Ethernet, but on the Raspberry PI with Raspbian
(Debian) Linux “Wheezy” v7.0 operating system the image
acquisition on Ethernet with the OpenCV function is not
working. The solution is to use external function, it can be
even used a Linux command like the “wget”, because the
image is already saved on the IP camera’s memory. This way
the image is transferred on the Raspberry PI’s memory card
and it can be loaded for image processing with OpenCV
functions.
Acquisitioning with USB web camera showed no errors
even with OpenCV functions executed on the Raspberry PI.
The only issue was to install the USB webcams on the
Raspberry PI, which was not so easy, because not all USB
Fig. 2. The robotic arm control application made in Python with OpenCV library on a Raspberry PI
691
webcams have Linux drivers. In this manner the Logitech
C270 was a good choice, because it had drivers created on
many operating systems.
The IP camera has the advantage that no driver is needed
to interface it with an operating system and can save the
acquired images on its internal memory; the only task is to
transfer the images over Ethernet to the PC or to the
development board. The big disadvantage, on some embedded
devices, is that the transferring of the saved images cannot be
done with OpenCV functions, only with external Linux
commands.
On Fig. 2 is shown the interface of the GUI, made in
Python, with the OpenCV libraries, which can control the
robotic arm on the RS-232 serial interface. The application is
created and running entirely on a Raspberry PI.
 The hardest part is that OpenCV had many dependencies
and they are linked mostly dynamically, this means that
OpenCV need to be installed on the Raspberry PI. Because the
model B has 700 MHz single core CPU, this can take quite a
long time, because the OpenCV is better to be installed from
source. It is hard to find “*.deb” precompiled packages of the
OpenCV libraries for a specific architecture, and if it can be
found, most of the time they are not working on some special
ARM platforms like the one from the Raspberry PI. The
compilation of the OpenCV takes more than 8 hours on
Raspberry PI model B and more than 4 hours on a Raspberry
PI 2, so when compiling it’s a good choice to take a long
brake. Sometimes is better to start it in de evening and it will
be done in the morning. After it’s done and it’s compiled with
all the GUI elements, JPEG or TIFF libraries, C/C++
programming language and Python scripting language, it will
work perfectly on a Raspberry PI.
On Fig. 2 the first there is the robotic arm image with the
guide lines and circles created for computing the robotic arm’s
position in space.
The second image is the HSV (Hue, Saturation and Value)
filter.
The next four images are the color recognition algorithm
for each color. On the joints there were glued bottle stoppers
of different colors. Each color was recognized using the
contour detection algorithm used over the color filtering
algorithm. The color filtering is done by setting the minimum
and maximum values of the HSV filter.
IV. CONCLUSION
As it was presented, a robotic arm control application was
created, using only simple web cameras and a Raspberry PI.
The Raspberry PI is the card size, embedded, super computer,
with the power and the possibilities of a PC. It has a big
advantage over a PC, because the operating system is on a
memory card, so the user can have different operating system
on different memory cards and the operating systems can be
switched just by changing the memory card from the
Raspberry PI.
The Linux operating system gives many advantages, one
of them is that the applications from it can be ported on other
systems quite easily, maybe the only task is to recompile the
source code and in case of Python this is not necessary
because the script can run on any machine with Python
installed.
Further enhancements would be to port the system on
FPGA boards with Ubuntu Linux 12.04 LTS like the ZYBO or
ZedBoard. These boards have little bit better performances
like the Raspberry PI (700 MHz single core CPU), because
they have ARM Cortex-A9 Dual-core with frequencies
between 650 MHz – 1000 MHz. The final goal is to create an
ASIC form the FPGA implementations.
REFERENCES
[1] W.G. Hao, Y.Y. Leck, L.C. Hun, “6-DOF PC-Based Robotic Arm (PC-
ROBOARM) with efficient trajectory planning and speed control,” 4th
International Conference On Mechatronics, Kuala Lumpur, 2011, pp. 1-
7.
692
[2] W.Yang, J.H. Bae, Y. Oh, N.Y. Chong, B.J. You, S.R. Oh, “CPG based
self-adapting multi-DOF robotic arm control,” International Conference
on Intelligent Robots and Systems, Taipei, 2010, pp. 4236-4243.
[3] E. Oyama, T. Maeda, J.Q. Gan, E.M. Rosales, K.F. MacDorman, S.
Tachi, A. Agah, “Inverse kinematics learning for robotic arms with
fewer degrees of freedom by modular neural network systems,”
International Conference on Intelligent Robots and Systems, 2005, pp.
1791-1798.
[4] N. Ahuja, U.S. Banerjee, V.A. Darbhe, T.N. Mapara, A.D. Matkar, R.K.
Nirmal, S. Balagopalan, “Computer controlled robotic arm,” 16th IEEE
Symposium on Computer-Based Medical Systems, New York, 2003, pp.
361-366.
[5] M.H. Liyanage, N. Krouglicof, R. Gosine, “Design and control of a high
performance SCARA type robotic arm with rotary hydraulic actuators,”
Canadian Conference on Electrical and Computer Engineering, St.
John's, CA, 2009, pp. 827-832.
[6] M. Mariappan, T. Ganesan, M. Iftikhar, V. Ramu, B. Khoo, “A design
methodology of a flexible robotic arm vision system for OTOROB,”
International Conference on Mechanical and Electrical Technology,
Singapore, 2010, pp. 161-164.
[7] H. Guo-Shing, C. Xi-Sheng, C. Chung-Liang, “Development of dual
robotic arm system based on binocular vision,” International Automatic
Control Conference, Nantou, 2013, pp. 97-102.
[8] R. Szabó, A. Gontean, “Controlling a Robotic Arm in the 3D Space with
Stereo Vision,” 21th Telecommunications Forum, Belgrade, 2013, pp.
916-919.
[9] R. Szabó, A. Gontean, “Robotic arm control in 3D space using stereo
distance calculation,” International Conference on Development and
Application Systems, Suceava, 2014, pp. 50-56.
[10] R. Szabó, A. Gontean, “Remotely Commanding the Lynxmotion AL5
Type Robotic Arms,” 21th Telecommunications Forum, Belgrade, 2013,
pp. 889-892.
[11] R. Szabó, A. Gontean, “Creating a Programming Language for the AL5
Type Robotic Arms,” 36th International Conference on
Telecommunications and Signal Processing, Rome, 2013, pp. 62-65.
[12] R. Szabó, A. Gontean, “Full 3D Robotic Arm Control with Stereo
Cameras Made in LabVIEW,” Federated Conference on Computer
Science and Information Systems, Kraków, 2013, pp. 37-42.
[13] R. Szabo, A. Gontean, “Robotic Arm Control with Stereo Vision Made
in LabWindows/CVI,” 37th International Conference on
Telecommunications and Signal Processing, Berlin, 2014, pp. 635-639.
[14] M. Seelinger, E. Gonzalez-Galvan, M. Robinson, S. Skaar, “Towards a
robotic plasma spraying operation using vision,” IEEE Robotics &
Automation Magazine, vol. 5, issue 4, 1998, pp. 33-38, 49.
[15] R. Kelly, R. Carelli, O. Nasisi, B. Kuchen, F. Reyes, “Stable visual
servoing of camera-in-hand robotic systems,” IEEE/ASME Transactions
on Mechatronics, vol. 5, issue 1, 2000, pp. 39-48.
[16] V. Lippiello, F. Ruggiero, B. Siciliano, L. Villani, “Visual Grasp
Planning for Unknown Objects Using a Multifingered Robotic Hand”,
IEEE/ASME Transactions on Mechatronics, vol. 18, issue 3, 2013, pp.
1050-1059.
[17] M. Kazemi, K.K. Gupta, M. Mehrandezh, “Randomized Kinodynamic
Planning for Robust Visual Servoing”, IEEE Transactions on Robotics,
vol. 29, issue 5, 2013, pp. 1197-1211.
[18] R.T. Fomena, O. Tahri, F. Chaumette, “Distance-Based and Orientation-
Based Visual Servoing From Three Points”, IEEE Transactions on
Robotics, vol. 27, issue 2, 2011, pp. 256-267.
[19] N.C. Orger, T.B. Karyot, “A symmetrical robotic arm design approach
with stereo-vision ability for CubeSats,” 6th International Conference on
Recent Advances in Space Technologies, Istanbul, 2013, pp. 961-965.
[20] F. Medina, B. Nono, H. Banda, A. Rosales, “Classification of Solid
Objects with Defined Shapes Using Stereoscopic Vision and a Robotic
Arm,” Andean Region International Conference, Cuenca, 2012, pp. 226
[21] M. Puheim, M. Bundzel, L. Madarasz, “Forward control of robotic arm
using the information from stereo-vision tracking system,” 14th
International Symposium on Computational Intelligence and
Informatics, Budapest, 2013, pp. 57–62.