Wearable Computing Device with Gestural
Augmentation using LabVIEW
Eliyaz Mahammad1, Madhu Nakirekanti2, G.Bhaskar Phani Ram3
1,2,3Assistant Professor
Department of ECE
Vardhaman College of Engineering
Hyderabad, India
Abstract—Our motivation is from the analysis of Mr. Steve
Mann on computational imaging and wearable computing, that
brings out sixth sense technology. Wearable computing is the
start of, developing, designing, or using very small wearables
and sensor devices. This device lets us to move with the data by
simple hand gestures. The recognition of gestures are the basic
idea behind this device. The main aim of our project is to bring
digital and natural worlds nearer, virtually together. Our
project work uses a camera which captures the gestures,
process it using RT processor (myRIO) by using LabVIEW
software and makes the desired action to be done. The
application of our device includes
i. Gestural camera that permits us for taking images by
framing the finger gestures.
ii. The data can be copied from physical world, like a
paragraph on a printed paper with hand gesture and paste
them on a digital device, i.e., gesture based pick from a real
paper and drop on a digital computer screen, rather than
typing.
iii. Replacement of Physical keyboard by printed paper.
Keywords— Wearable Computing, Gesture Recognition,
LabVIEW, myRIO.
I. INTRODUCTION
The wearable computing device consists of “always turn
on computer combined with a control interface and display
(head mounted) and have become a practical possibility.
The Steve Mann who is the first adopters and advocates
of this form of usage of computer, wearables and arrived at
three basic properties. The first one deals with a computer
which is wearable and is not carried, and came to be a part
of the user; the second one states that it is easily controllable
by the user, not compulsorily involving conscious idea or
attempt and, third one is, it runs in real time and it is
constantly active and the user can be interacted at any time.
By making use of these definitions it was possible to
recognize early applications of wearable computing [1-3]
retrospectively. These involved the shoe mounted roulette
wheel prediction system by Shannon and throp, realized in
1960, and well developed and pre owned by the
'Eudaemons' and commercialized by Keith and others by
1983. Kaith again designed camera which was belt
mounted that was able to relay card, as it would be dealt, to
a satellite receiver in a pickup truck which was in the
parking lot, where an associator read the video picture and
then player has been signaled at the table with the
information he required to play his hand.
Here our project requires a simple camera that captures
the gestures and these will be processed by the RT processor
which uses LabVIEW software and makes the required
action. LabVIEW software is developed by National
Instruments and it is a visual programming language
XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE
consists of system design platform and development
environment.The RT processor that was used in this project
is myRIO from National instruments. NI myRIO is a real
time embedded evaluation board which uses LabVIEW
software. It is used to construct applications that uses its
onboard microprocessor and FPGA.
II. APPLICATIONS
Wearable Computers find various applications in
differentfields like industrial, military and medical fields etc
A. Industrial Applications
The accurate and quick availability of information which
is complex to the working person in the field, or in a
workplace (non- office), has become an important
consideration of many organizations after the establishment
of computerized storage in the 1950's. While this would be
obtained with the use of handheld equipment, large number
of workers utilize one or both hands for carrying out their
assigned tasks, and with their piece of work they have to
maintain eye contact. The wearable computers available with
a head-mounted display and hands free interface will provide
a good solution for these workers.
B. Military Applications
The enforcement agencies associated military
organizations quickly acknowledges the importance of
wearable devices to a soldier. To agitate terrible situations,
the sensible data assisting with differentiating between
friendly and opposed forces, and certain giving approaches,
are often accessed with wearable devices. Naturally a lot of
this analysis was classified as confidential, but the
collaboration with non-military researchers would be found
within the united states, U.K, Singapore and Australia. The
main intention is, during a combat-effective, cohesive
system, the technology and the soldier needs to be merged.
C. Medical Applications
The applications represented earlier have used position
sensing technology to help in numerous varieties of tasks. the
data of wherever the address of user clearly provides the
support for several wearable styles. wearable devices may
even be designed to observe well-being and activity of the
user. Wearables have the potential to observe health to help
with rising performance e.g. sports; prevention and detection
of sickness through diagnosis; and even treatment, although
this typically involves some procedure that is invasive.
III. LITERATURE SURVEY
The development of this technology traced back to the
1990’s. During this period, The Steve Mann developed a
wearable computer, he is known as the father of sixth sense
technology[4-6]. This technology consists of a light source
joined with simple camera and was evolved in 1997. But the
name was first published in 2001 as Sixth Sense. He
developed this technology as a simple camera with a neck
worn projector. Bryce Kellogy implemented a overall system
and it becomes an effective innovative and very important
step towards the gesture recognition development for
mankind. The next development of this technology was done
by Pranav Mistry from India. He was an research assistant in
MIT media Lab. By using this technology Mistry
implemented new applications coined the term “Wear Ur
World‟ (wow). The earlier prototype has one limitation
because it has a helmet with a big projector surrounded on it
which causes the problem, if the person projecting the data
on wall and suddenly turns to talk with someone, then the
information will be projected over the person from the
projector. By using new neck worn prudent prototype the
above limitation will be avoided. The initial prototype had an
normal web camera. Pranav Mistry had this device around
his neck, and four coloured markers (red, green, blue and
yellow) placed on both the hands. After an intensive analysis,
the authors found the term “wearable computer” has been
outlined in many ways. for example, As per Massachusetts
Institute of Technology wearable computing laboratory,
wearable pc could be a device that will be worn very much
like “eyeglasses or clothing”, and act with the user that
depends on the scenario. With features like personal WAN’s,
non conspicuous input devices, and alert displays, the
wearable pc would behave as an intelligent assistant..
According to NASA used the term Body Wearable
Computer (BWC), which has battery-powered computer
system worn on the user's body and the unit is implemented
for mobile and mainly hands-free operations, often uses
speech input and head-mounted displays.
The Gesture Computing could be a best innovation that
enables hand development and outward appearances as data
controls. Various gadgets are used now-a-days for capturing
pictures and putting away in their mass reposition gadgets
like processed cam with memory cards or basic memory, cell
phones with built-in memory and memorycards, and so
forth.This project proposes the implementation and progress
of the input device like keyboard and paint application with
the help of hand gestures. Any application can be controlled
which brings out a path to human computer interaction by
using these hand gestures.
IV. IMPACT, RELEVANCE AND NOVELTY
The touch less sensing market & gesture recognition is
expected to grow from USD 5.15 Billion in 2014 to USD
23.55 Billion by 2020. Most of the companies in the touch
less sensing market & gesture recognition include Intel
(U.S.), Prime Sense (Israel), Cross Match Technologies, Inc.
(U.S.), Microchip Technology Inc. (U.S.), Qualcomm Inc.
(U.S.) and more AR technology is the most emerging
technology now a days.
Our experience with LabVIEW environment (including
software and hardware) makes us to believe strongly that our
project with some upgrades can make wonders. By this
project it is possible to play games on personal computer
with gestures and any non-touch monitor can also be turned
into touch monitor We can even replace the mouse of
computer and use it by hand gestures The companies
working on AR technology using some other software
procedure to make their products but working with LabView
make AR technology to more efficient .Our project makes
life of blind people and physically handicapped as it uses
image processing techniques to control the devices interfaced
with it. Every individual can adopt this technology and
configure it based on the problems they are facing. Like our
project as a product for physically handicapped will help
them to control the device configured with it with simple
hand gestures .Our project as a product serves as both
Business 2 Business and Business 2 Consumer product. It is
the self-selling product.
V. SIXTH SENSE DEVICE
Sixth sense is the capability to know the fine-dimension
or the hidden world. It additionally incorporates our
capability to understand the fragile cause and impact
relationship behind many events that's on the far side
understanding of the intellect. We’ve got a liability to
acknowledge visible world through the five physical senses
(i.e. sight, smell, touch, taste, and sound), our mind, and our
intellect. Once it involves the unseen world or the subtle-
world, we've got an inclination to feel it through the five
subtle-senses, the subtle-mind and thus the subtle-intellect –
and of popularly referred to as our intuition. We tend to use
our five natural senses to know any data; that data helps us
build choices and choose the right actions to take. However
in all probability the most useful information which is able to
facilitate us to form the right decision is not naturally
perceivable with our five senses, notably the data,
knowledge, and data that group has accumulated regarding
everything and that is increasingly offered on-line. Intuition
technology bridges this gap, develops intangible digital data
out into the tangible world, and allowing us to maneuver
with this information via hand gestures.
The sixth sense Technology uses the subsequent devices:
A. Camera
It selects the motion of the colored markers worn at the
fingertips and user hand gestures are tracked. The camera
acknowledges images, individuals, gestures, photographs,
that person makes with his hand. The digicam then sends
these records to the laptop or pc for processing. The digital
camera plays a fundamental interface device between laptop
and the physical world.
Fig. 1. Web cam
B. Coloured Marker
These color markers placed at the tip of user fingers.
Marking the user fingers with yellow, red, blue and green
colored tape makes the digital camera to acknowledge the
hand air gestures. The movement of the color markers set at
the tip of the hand is recognized and several other hand
gestures are fed relying upon the mixture of colors present
within the laptop computer in the form of preloaded
information.
Fig.2. Colour markers
C. Laptop or PC
The computer or laptop is going to be used for
implementing and running the code written on the simulation
tools for the execution of the idea of image processing. The
camera tracks different hand gestures carried out by the color
marker and therefore applied to the PC or Laptop for further
operations of the various applications.
to communicate naturally without the need of any
mechanical instruments
C. Computer Vision
The software package tracks the user gestures using
computer-vision based mostly algorithms. pc vision is that
the inverse of computer graphics. Here image data is
produced by the computer graphics from three dimensional
models, while computer vision often generates three
dimensional structures from image information
VII. DESIGN AND IMPLEMENTATION
A. Block Diagram
The Block diagram of the project shown in Fig. 4 gives
the functioning of the project flow. As the user performs the
specific actions before the camera using the color markers
and the camera grabs the images and process them as per the
code dumped in the myRIO. The camera connected to
myRIO is mounted onto the body like pendent or could be
worn attaching to the cap.

Camera recognizes
the user performed
User performing gestures and sends to
predefined gestures myRIO for processing
using colour markers

Fig. 3. Laptop

VI. TECHNOLOGIIES BASED ON SIXTH SENSE TECHNOLOGY

Sixth Sense technology uses a spread approach for myRIO processes the session
computation and makes the digital exposure of our human recognized by camera
lives additional interactive, intuitive, and, above all, a lot of Displays and stores the according to the code
natural. we must always not consider it on an individual data sent by myRIO developed and dumped in
wirelessly RT processor
basis. it's a lot of complicated technology squeezed into an
easy transportable device. Once we come with Fig.4. Block Diagram
implementation, we might get prompt, pertinent, visual
information projected on any object we pick up or the
technology will be interacted will be primarily depends on B. Components
gesture recognition, augmented reality[7-9] and computer The Various components used in this project have been
vision algorithm. discussed below.
A. Augmented Reality
1) Battery
Augmented reality[10-12] may be a phrase for a live 8AA batteries with barrel type connector at 1.25Volts
direct and indirect vision of a true world environment where each which will be used to power a 14Watts myRIO system
the elements are augmented by virtual system established with webcam for 1.4 hours.
imaginary.
B. Gesture Recognition 2) Colour Markers
Four Color markers of colors (Red, Yellow, Green, Blue)
Gesture recognition provides a path for computers to start
used for color detection and are detectable by camera and
out with understanding physical body movements, therefore
processed by vision assistant system.
developing a bridge between persons and machines and so
3) NI myRIO
even graphical user interfaces, that still limit the majority of
NI myRio hardware with code deployed in it for image
input to keyboard and mouse. Gesture recognition makes
processing and image acquisition is used as an embedded
human beings for interfacing with the device and allows us
platform which communicates with laptop wirelessly.
myRIO is a real-time embedded evaluation board developed
by National Instruments. The structure of myRIO is shown in
Fig. 5 below. It is used for the development of applications
which uses its FPGA and µP. It requires LabVIEW. The
National Instruments myRIO-1900 is a portable
reconfigurable Input output device which will be used for
designing and controlling and mechatronics and robotic
systems.
botic systems.

Fig.7. Connection Diagram

VIII. RESULTS
Fig.5. NI myRIO
A. Gestural Camera
The ni myRIO-1900 has audio, and power output pins,
analog input (AI), analog output (A0), digital input and
output (DIO), in a very dense embedded instrument. The ni
myRIO-1900 connects to a laptop over wireless 802.11b and
USB as shown in Fig. 6below

Fig.8. Block Diagram of Gestural Camera

The camera detects the gestures performed in front it

Fig.6. NI myRIO Internal Block Diagram
4) Camera
Logitech 5V, 500 mA webcam which takes power from
NI myRIO will be used for image acquisition.
using colour markers and checks the condition that if the
distance between the coordinates of two colours (blue and
yellow)&(red and green)is in between -100 to 100 but not
zero. The image logging of the Vision Acquisition is enabled
and the image is captured by the web camera connected to
myRIO and saves the single image into the image log folder.
The block diagram and front panel of gestural Camera is
shown in Fig. 8 and Fig. 9 respectively. The image showing
the detected colour templates is shown in Fig.10

5) Laptop
Laptop is used for storing data and displaying data
processed from myRIO which communicates with myRIO
wirelessly.
6) Connection Diagram
A Battery set with barrel type connector connected to
myRIO adaptor port will supply power to myRIO and the
web camera connected to it. Web camera will be connected
to the USB device port of myRIO. Laptop and myRIO are
connected wirelessly. The connection diagram is shown in
Fig. 7 Fig.9. Front Panel of Gestural Camera
 Fig.10. Image showing the detected colour templates

B. Pick and Place in to a digital screen Fig.13. Front Panel of Pick and Place Feature
Picking and placing into digital screen includes a gesture
to pick the image from the natural world. The camera
captures the image and the selected part of image is trimmed
by IMAQ Extract 2. The LabVIEW icon for IMAQ Extract 2
is shown in Fig. 11

Fig.14. Image showing the overlays over the number templates and
red colour template
Fig.11. IMAQ Extract 2

The selected part from the real world paper will be

recognized by the coordinates of the colour markers and
trimmed by the IMAQ Extract 2 palette. These are fed as
input to the optional rectangle by which the selected part is
trimmed. The function write image acquires the single image
from the image output and the image will be written into the Fig.15.Tthe image showing OCR detecting all the characters in the
memory when the two colours (red &yellow) shows the area keypad
to be picked. IMAQ Extract extracts the image from real
Here the user uses a keyboard patterned paper. He would
world by considering the ROI as the coordinates of those two
show the alphabet to be typed by his finger and the character
templates. The block diagram and front panel of Picking and
is recognized by OCR by considering ROI slightly above to
Placing feature is shown in Fig. 12 and Fig. 13 respectively.
that of the colour template. The data typed is copied into a
word file programmatically. The image showing OCR
detecting all the characters in the keypad is shown in Fig. 15
The front panel of virtual keyboard is shown in Fig. 16

Fig.12. Block Diagram of Pick and Place Feature

C. Repalcement of Physical keyboard in to printed one

The vision assistant detects the number templates and the Fig.16. Front Panel of Virtual Keyboard
colour template as shown in Fig. 14 below
 CONCLUSION
This technology will be used as a replacement of the five
senses for disabled peoples. this will give simple control over
machineries in industry. This technology has various
applications for different developers based on how the
developer thinks and what he needs. By making use of
gesture movements, speech integrated circuits, LabVIEW
software and hardware have made sixth sense technology an
emerging innovation. It provides us a smooth access to
information that may help us to make crucial decisions. The
masterstroke here is that Sixth Sense technology detects the
objects around and make us to use that information in the
way we need and projects that data as well, all this in the
most easiest of the different ways. The well chosen
awareness of this technology will project towards the further
development and usefulness of this technology, which in-
turn will aid in acquiring information and operating any type
of function practically at any time. And this can be achieved
simply by using gestures and commands.
[7] Bass, L., Kasabach, C., Martin, R., Siewiorek, D., smailagic, A. and
Stivoric, J., “The Design of a Wearable Computer”, Proceedings of
CHI97, (1997) pp 139-146.
[8] Campbell, N.W, Mackeown, W.P., Thomas, B.T. and Troscianko, T.,
“Automatic Interpretation of Outdoor Scenes”, British Machine
Vision Conference, September (1995).
[9] Caudell, T.P. and Mizell, D.W., “Augmented Reality: An Application
of Heads-Up Display Technology to Manual Manufacturing
Processes”, Proceedings, IEEE HawaiiInternational Conference on
Systems Sciences, January (1992), pp 659-669.
[10] Doyle (Boland) E.A., Weinzimer, S.A., Steffen, A.T., Ahern, J.H.,
Vincent, M. andTamborlane, W.V.. “A Randomized, Prospective
Trial Comparing the Efficacy ofContinuous Subcutaneous Insulin
Infusion with Multiple Daily Injection Using Insulin Glargine”.
Diabetes Care, 27, (2004), pp 1554-8.
[11] Duchamp, D., Steven, K. F. and Gerald Jr. Q. M., “Software
Technology for Wireless Mobile Computing”, IEEE Network
Magazine, 12(18), (1991) p.218.
[12] Feiner, S., MacIntyre, B. and Seligmann, D., “Knowledge based
augmented reality,Communications of the ACM”, 36(7), (1993), pp
53-62.

References
[1] Prof. D.S. Patil, M. Shahak Patil, “Sixth Sense Technology- A New
Innovation”, International Journal On recent and Innovation trends in
computing and Communication, Volume-2, issue-5, pp. 1200-1204,
ISSN: 2321-8169.
[2] Aakansha Yeole, S. Chitraby using V.sriraman, PoojaShinde,
Priyanka Sadhwani, “Virtual Keyboard", International Journal of
Science Engineering and Technology Research, vol.3, issue 3,March
2014 .
[3] T. Starner, "The Challenges of Wearable Computing", IEEE Micro,
21, pp. 54-67, July 2001.
[4] Use your hand as a 3-D mouse or relative orientation from extended
sequences of sparse point and line correspondences using the affine
trifocal tensor. In proceedings 5th European Conference on Computer .
Vision, 1998, pages 141-157.
[5] O. Castillo, O. and P. Melin, "A New Approach for Plant Monitoring
using Type-2 Fuzzy Logic and Fractal Theory", International Journal
of General Systems, Taylor and Francis, Vol. 33, 2004, pp. 305-319.
[6] Bakay, R.E., and Kennedy, P.R. In B. Siuru, “A Brain/Computer
Interface, ElectronicsNow”, March, (1999), pp 55-56.