GESTURE CONTROLLED ROBOTIC ARM
1. Shivashish Bohra 2 . Chaya K
Student, Department of ME Student, Department of EEE
Sapthagiri College of Sapthagiri College of
Engineering,Bengaluru, India Engineering ,Bengaluru, India
shivashishsarvr@gmail.com chayack1997@gmail.com
5. Irene Jacob
Student, Department of EEE
Sapthagiri College of
Engineering , Bengaluru, India
ireneiris0397@gmail.com
Abstract—This paper focuses on Design and control of
a robotic arm with human hand gestures using computer
vision techniques. Computer Vision is a field of deep
learning that enables machines to see, identify and process
images. Use of OpenCV (python) for computer vision
provides a powerful environment for learning since it is
easy to use compared to other techniques. In this
technology, a camera reads the movement of hand and
communicates with the computer that uses gestures as
input to the control devices.
Keywords—Robotic arm, hand gestures, Computer
Vision, OpenCV, powerful environment, communicates,
control devices.
I. INTRODUCTION
Robotics is an inter-disciplinary branch of engineering
which deals with the design, construction, operation, and
use of robots , as well as computer systems for their
control, sensory feedback, and information processing.
Computer vision provides basis for applications like
image analysis using automation which enables us to
determine any object or activity in the given image. Some
of the task performed using computer vision is
recognizing simple geometric object, analyzing printed or
hand-written characters, identifying human faces and
hand gestures.
The efficiency of the control of robotic arm is achieved
using gesture recognition techniques. The gesture images
captured can be processed by computer vision techniques
[1]. Gesture recognition using Kinect sensor in various
fields such as control of wheelchair movements and also
3. Arpitha Srinivasa Murthy 4. Apoorva K G
Student, Department of EEE Student, Department of EEE
Sapthagiri College of Sapthagiri College of
Engineering ,Bengaluru, India Engineering ,Bengaluru, India
arpithasrinivas1997@gmail.com apoorvakg1998@gmail.com
pick and place objects. Microcontroller is used to control
the rotation of wheels [2]. Kinect sensor traces the
skeletal movements to provide gestures for the robotic
arm prototype. Bluetooth is used for wireless transmission
of signals to the robotic arm [3]. The use of gesture
control is also used in remote rescue operation where no
humans can go and has the ability to become a part of
massive rescue operations [4]. The gesture recognition
involving counting of fingertips in the gesture can be
implemented [5]. The use of microcontroller to control
the robotic arm is less efficient as compared to Arduino
[6]. Robotic arm can be a great boon to the handicapped
as prosthetic arm [7]. The design of glove-based arm
using neural network to detect objects and do pick and
place operations [8] or to navigate the arm in space with
appropriate gestures [9].
II. PROPOSED SYSTEM
We provide a system in which the user can navigate the
wireless robot in the environment using various gesture
commands.
Camera is used to capture real time hand gestures to
generate commands for the robot. Gesture is taken as an
input and processed using Image processing.
Arduino takes the signal from camera as input and
generates output signals. This output signal generation
depends on the gesture input, for every possible gesture
input, different output signal is generated. Servo motor
takes digital signals as the input from the Arduino .Once a
command signal is given to the robotic arm, it replicates
the gesture
.
 GESTURES

3. Thresholding process
CAMERA
Image thresholding is an important intermediary step
for image processing pipelines. Thresholding can help
POWER ARDUINO UNO us to remove lighter or darker regions and contours of
SUPPLY images. Grabbing all pixels in the gray image greater
than 225 and setting them to 0 (black) which
SERVO MOTOR corresponds to the background of the image.
Thresholding is the process of assigning pixel
intensities to 0’s and 1’s based a particular threshold
level so that our object of interest alone is captured
ROBOTIC ARM
from an image.
Fig 1. BLOCK DIAGRAM

III. GESTURE RECOGNITION USING OPENCV

AND PYTHON

1. Capturing Frames

The input frames(RGB format) is captured by camera Fig 3.3. Threshold process
and it is converted into gray scale image and the
region of interest is extracted as shown in fig 1. 4. Drawing contours ,finding convex hull
and convexity defects

In computer vision, contours are drawn by scanning

the binary image from left to right and from top to
bottom to find first contour pixel and then scanning
clockwise till the next pixel value becomes 1.
The largest contour (fingertips) is extracted and then
convex hull, a convex set enclosing the hand region
is drawn. The green line in fig 5 is the convex hull
Fig 3.1. Input frame
and the blue dots are the defect points. Depending on
the number of defect points, the gesture is identified.
2. Blur frame

For blurring the image using Gaussian blur function,

the gray scale image is convolved with a Gaussian
filter, a low-pass filter that removes high-frequency
components.

Fig 3.4. Convex hull and Convexity defects

5. Counting the fingers

The no of fingers is identified based on the distance

Fig 3.2. Blur image between the centre and the tip of finge0 r(radius).
III. INTERFACING PYTHON WITH ARDUINO IDE
Interfacing python with Arduino is done by pyFirmata
which is basically a prebuilt library package of python
program which can be installed to allow serial
communication between a python script on any computer
and an Arduino.
IV. HARDWARE IMPLEMENTATION
In this technology, camera reads the gesture and
communicates with the computer that uses images as
input.
The image is processed by the system and the processed
data of the thresholded image is converted into the angle of
rotation of servo motors by sending command to the
Arduino.
The Arduino sends the required PWM to the servo motors
for appropriate movement of the fingers connected to it.
Fig 4.1. Variable pulse width control of servo motor
• The servo motor works by using pulse width modulation
for control.
• A standard servo motor can rotate from 0 degrees to 180
degrees.
• Varying the pulse width between 1ms and 2ms varies the
servo position between 0° and 180°.
V. CONCLUSION
Gesture recognition helps computers to begin to
understand human body language, thus building a
richer bridge between machines and humans than
primitive text user interfaces or even GUIs (graphical
user interfaces), which still limit the majority of input
to keyboard and mouse.
The control of robotic arm by interfacing the
Arduino to Bluetooth can enable distant
communication. The efficient operation at poor
lighting conditions should be accomplished. Also the
use of sensors to detect obstacles can be done.
VI. FUTURE SCOPE
Robotic Arms has a wide scope of development. In
the near future the arms will be able to perform every
task as humans and in much better way. Brain
Computer Interface (BCI) is an immerging field of
research. BCI can be used to acquire signals from the
human brain and control the arm. The system can
work in the same way as human arm. A person who
may have lost his hand in any accident can resume
his life like previous by such artificial arms. Robotic
arms are versatile and have enormous ways of
implementations.
VI. REFERENCES
1. Prutha Atre, Sahil Bhagat, Nevil Pooniwala, Payal
shah, ’ Efficient and feasible Gesture Controlled
Robotic Arm’-IEEE(ICICCS),2018
2. M.Ababneh,H.Sha’ban,D.Khader,H.Mahameed,
M.AlQudimat,
‘ Gesture Controlled Mobile Robotic Arm for Elderly
and Wheelchair people Assistance Using Kinect
Sensor’ -15th international multi-conference on
SSD,2018
3. Memon Md. Farhan Md. Fareed, Shaikh Bilal Ahmed
Anees,‘Gesture based wireless single armed robot in
cartesian 3D space using Kinect’-5th international
conference on communication system and network
technologies,2015
4. Pantha Protim Sarker, Farihal Abedin and Farshina
Nazrul Shimim.‘R3Arm: Gesture controlled Robotic
Arm for Remote Rescue operation’-IEEE(R10-
HTC),2017
5. Ibrahim Baran CELIK, Dr. Mehmet KUNTALP.
‘Development of a Robotic-Arm Controller by using
Hand Gesture Recognition’IEEE 2012.
6. Ganesh Choudhary B, Chethan Ram B ‘Real time robotic
arm control using hand gestures’-2nd international
conference on Machine learning and computing, 2010.
7. Nicholas Bonini, Nithya Iyer, David Kim, Katherine
Mathison, Laurewellons.‘Robotic hand in motion using
Arduino-controlled servos’New Jersey governor’s school
of engineering and technology 2014.
8. Asif Shahriyar Sushmit,Fariha Musharrat Haque,Md.
Shahriar, Shaikh Al Mahmud Bhuiyan, M.A.Rashid
Sarkar.‘Design of a Gesture Controlled Robotic Gripper
Arm using Neural Networks-IEEE(ICPCSI),2017.
9. Harish Kumar Kaura, Vipul Honrao, sayali Patil, Pravish
Shetty..‘Gesture controlled robot using image Processing’
(IJARAI)Vol.2,No.5,2013