CHAPTER 1

INTRODUCTION
The 21st century depends entirely on technology. The importance of technology is
everywhere such as in health, finance, education, corporate world etc. The technology which
we use is constantly evolving due to which innovations takes place very frequently in today’s
world. Thus the size of computer devices are getting smaller day by day. One such innovation
is Sixth Sense Technology. Steve Mann, is regarded as the brain behind this technology, who
made the neck worn computers in 1997.
Later on this work was continued by Pranav Mistry, a PhD student at the MIT Media
Lab, who is now Vice President of Research at Samsung. The information around humans
regarding touch, vision, taste, smell, hearing is perceived by using the five senses. But the most
important information, i.e. data, which is available online through internet, is not naturally
recognized by human senses, withsix sense technology such information will be readily
available to the users as the gap between the digital world and the physical world is eradicated.
The Sixth Sense technology will revolutionize the world in all aspects, the tasks which take
few minutes today will be performed in few milliseconds due to sixth sense technology.
This technology will give humans the freedom to use computer anywhere and
everywhere, the entire world will be a computer due to this technology. The device is
comprised of components such as data projector, camera, mobile component, color markers,
mirrors and microphone. All the components have certain functions and they work in
collaboration with each other.
1.1 HISTORY
The history of “Sixth sense technology” goes back into 1990’s when Steve Mann first
attempted to propose a neck worn projector and a camera combination.
The idea of implementing computer technologies to daily tasks as ours Sixth Sense was
future developed by Pranav Mistry who also appears to be an MIT student as well as Steve
Mann. The first prototype of Sixth sense technology was actually bigger than what it looks like
today and it was not working properly to use in a daily life. In an article called The Sixth Sense
technology Arjun K.R says that “They started with a larger projector that was mounted on a
helmet. But that proved cumbersome if someone was projecting data on to a wall then turned
to speak to friend-the data would project on the friend’s face”(Arjun K.R 2011).Therefore,
Mistry came up with different and more convenient device which is a neck worn portable
camera that allows users more space for their daily actions.
On the other hand, Mistry first tried out his idea on a simple computer mouse. First, he
put two rollers into one mouse and see if he could obtain data and guide the movements of the
mouse. Two rollers did not work properly so he decided to use four rollers and see if it could
work better. Four rollers gave him the idea that he could use the same idea on fingers and that’s
what he next moved on to. The fig 1.1 below shows how four rolls were tried out in Mistry’s
experiment

Figure 1.1: Four mouse rollers tried in the experiment

1.2 APPLICATIONS
The Sixth Sense prototype implements several applications that demonstrate the
usefulness, viability and viability of the system. The Sixth Sense device has a huge number of
applications. The following are few of the applications of Sixth Sense Technology.

 Call using palm as dialer:

You can use the Sixth Sense to project a keypad onto your hand, and then use
that virtual keypad to make a call. It gives the provision of using the hand as a screen
onto which the dialer is projected, using which an individual can make a call. Calling a
number also will not be a great task with the introduction of Sixth Sense Technology.
No mobile device will be required, just type in the number with your palm acting as the
virtual keypad. The keys will come up on the fingers. The fingers of the other hand will
then be used to key in the number and call as shown in fig.1.2

Fig1.2: Palm as dialer

 Capturing Photos using Fingers:
The burden of carrying a camera and keeping it safe and sound on a trip can
now be avoided by simply using the Sixth sense technology. The sixth sense computer
works as a camera. By making a square with our fingers and specifying the image we
want to frame, the camera takes the photo.

Fig1.3: Capture Pictures Using Frame Gesture

 Get product information:

Sixth Sense uses image recognition or marker technology to recognize products

you pick up, and then feeds you information on those products. For example, if you're
trying to shop "green" and are looking for paper towels with the least amount of bleach
in them, the system will scan the product you pick up of the shelf and give you guidance
on whether this product is a good choice for you as shown in the fig 1.4

Fig1.4: Product information

 Reviewing Flight Status:
Any background can be used to project a screen, on which the status of flight
can be checked by placing the ticket in front of this technology. The system will
recognize your boarding pass and let you know whether your flight is on time and if the
gate has changed as shown in fig 1.5

Fig1.5: Flight updates

  Time without watch:
Time can be viewed merely by drawing a circle on the wrist instead of
physically wearing a wristwatch. . The computer tracks the red marker cap or piece of
tape, recognizes the gesture, and instructs the projector to ash the image of a watch onto
his wrist.

Fig1.6: Wrist Watch

 Video newspaper:
It determines the news headline and the displays the appropriate video. The Sixth Sense
system also augments physical objects the user is interacting with by projecting more
information about these objects projected on them. For example, a newspaper can show live
video news or dynamic information can be provided on a regular piece of paper as show in fig
1.5. Thus a piece of paper turns into a video display.

Fig1.7: Video in newspaper

 Zoom in Zoom out:
It helps to view images and related things in the way one wants to see.

Fig1.8: Zoom in and out

  Feed information on people

Figure 1.9: Information on people

Sixth Sense also is capable of "a more controversial use. When you go out and
meet someone, projecting relevant information such as what they do, where they work,
and also it could display tags about the person floating on their shirt as shown in fig
1.9 It could be handy if it displayed their facebook relationship status so that you knew
not to waste your time.

• Get book information

Figure 1.10: Book information

Sixth Sense uses image recognition or marker technology to recognize products you
pick up, and then feeds you information on books as shown in fig 1.10. The system can project
Amazon ratings on that book, as well as reviews and other relevant information
Chapter 2
SYSTEM DESIGN

2.1 System Architecture

Figure 2.1: Six Senses

Sixth Sense in scientific (or non-scientific) terms is defined as Extra Sensory

Perception or in short ESP. It involves the reception of information not gained through any of
the five senses. Nor is it taken from any experiences from the past or known. Sixth Sense
aims to more seamlessly integrate online information and tech into everyday life. By making
available information needed for decision-making beyond what we have access to with our
five senses, it effectively gives users a sixth sense The fig 2.1 shows the additional sixth
sense to our five sense.

2.2 Earlier Sixth Sense Prototype

Figure 2.2: Earlier Device

Maes MIT group, which includes seven graduate students, were thinking about how a
person could be more integrated into the world around them and access information without
having to do something like take out a phone as shown in fig 2.2. They initially produced a
wristband that would read a Radio Frequency Identification tag to know, for example, which
book a user is holding in a store. They also had a ring that used infrared to communicate by
beacon to super-market smart shelves to give you information about products. As we grab a
package of macaroni, the ring would glow red or green to tell us if the product was organic or
free of peanut traces whatever criteria we program into the system.They wanted to make
information more useful to people in real time with minimal effort in a way that doesn’t require
any behavior changes. The wristband was getting close, but we still had to take out our cell
phone to look at the information.

They when they struck on the idea of accessing information from the internet and
projecting it. So someone wearing the wristband could pick up a paperback in the bookstore
and immediately call up reviews about the book, projecting them onto a surface in the store or
doing a keyword search through the book by accessing digitized pages on Amazon or Google
books. They started with a larger projector that was mounted on a helmet. But that proved
cumbersome if someone was projecting data onto a wall then turned to speak to friend the data
would project on the friends face.

2.3 Recent Prototype

Figure 2.3: Present Device

Now they have switched to a smaller projector and created the pendant prototype to be
worn around the neck. The Sixth Sense prototype is composed of a pocket projector, a mirror
and a camera. The hardware components are coupled in a pendant-like mobile wearable device.
Both the projector and the camera are connected to the mobile computing device in the users
pocket. We can very well consider the Sixth Sense Technology as a blend of the computer and
the cell phone. It works as the device associated to it is hanged around the neck of a person and
thus the projection starts by means of the micro projector attached to the device. Therefore, in
course, you turn out to be a moving computer in yourself and the fingers act like a mouse and
a keyboard. The prototype was built from an ordinary webcam and a battery-powered 3M
projector, with an attached mirror all connected to an internetenabled mobile phone.
The setup, which costs less than $350, allows the user to project information from the phone
onto any surface walls, the body of another person or even your hand. Mistry wore the device
on a lanyard around his neck, and colored Magic Marker caps on four fingers as shown in fig
2.3 (red, blue, green and yellow) helped the camera distinguish the four fingers and recognize
his hand gestures with software that Mistry created.

2.4 Advantages

 The digital information and its objects are integrated into the physical world by using the Sixth
sense interface, thereby making the entire world as our own computer.
 Sixth Sense makes machines like computers to adapt to human needs and not the other way round.
 Hand gestures are used to communicate with digital information, multi-touch and multi-user
interaction are also supported.
 Data from machine is directly accessed into real time. It is open source and it is cost effective and
map idea can be minded anywhere.
 Our relevant information is provided by the gesturecontrolled wearable computing device that
manipulates any surface into a display.
 It is portable
 Comparatively easy to carry as can be worn in our neck
 Even a naïve man with little or no knowledge of mouse and computer can use this device.
 Need to carry a camera no longer persist.
 The cost that arose for the making of the sixth sense technology proto type is exceptionally low.
A basic sixth sense device sums up to $300.

2.5 Disadvantages

 Excessive use of a technology can cause addiction and can hamper social life as well. It
will in-turn diminish humanness.
 Exposure of rays on surfaces like human arms can lead to health problems.
 This technology will affect the hardware market and will result in less revenue being
generated and lower the cash inflow.
 It can affect the vision of the user because of its peculiar and better use at night time and
in dark areas as compared to mornings and bright places.
CHAPTER 3
IMPLEMENTATION
3.1 Components
The main components of Sixth Sense Technology are as follows:
 Camera
 Color Markers
 Mobile Component
 Projector
 Mirror
 Microphone

3.1.1 Camera
This component is present at the head of the Sixth Sense Technology pendant. The
function of this camera is just like that of a human eye, the only difference being it will provide
digital information. This camera is capable of capturing the images that falls within its vision
and also it provides information about the objects present in front of it. The hand gestures
performed by the user can be tracked with this component. Additionally, facial recognition is
supported by this device. The information collected by camera is processed by the connected
mobile component.

Fig 3.1: Camera

3.1.2 Color Markers
This technology makes use of four color markers: red marker, green marker, blue
marker and yellow marker. The user can wear these markers at the tip of their fingers which
helps the camera to track hand gestures. These gestures can perform various tasks such as
painting, taking a picture and many more.
 Fig 3.2: Color Markers
3.1.3 Mobile Component
One of the most important components of this technology is the mobile module. The
function of this component can be carried out by any device such as smartphone, personal
digital assistant, laptop etc. as long as it is mobile and web-enabled. It is a processing engine
that processes the data obtained from camera and sends the output to projector.

Fig 3.3: Mobile Component

3.1.4 Projector
The projector is basically an output gadget which is used to display any information
provided by the mobile component. Projector can project information on any kind of physical
surface such as a wall, palm of user’s hand, a paper etc. and the user can interact with it.

Fig 3.4: Projector

3.1.5 Mirror
Mirror is placed just below the projector. The projector can project the information in
any direction with the help of a mirror which can be tilted in any direction as per the user’s
requirement.
 Fig 3.5: Mirror
3.1.6 Microphone

Fig 3.6: Microphone Clip

This component is used only when the projection is to be done on a piece of paper.
Microphone in form of a clip is affixed on that piece of paper. The information written by user
on that paper is recognized by the camera by interpreting the sound of user’s touch. The camera
is also capable of detecting the motion of the paper held in hand.
3.2 Working
Sixth sense device analyses whatever the user is watching and it augments the surfaces
and physical objects user is interacting with. A number of standard gadgets are connected
including a mini projector, camera, mobile phone, microphone and a mirror. The mobile phone
acts as a computer and our connection to web where all the information is stored.

Fig 3.6: Working of the Sixth Sense Technology

The components in the device are managed by a mobile phone device carried by the
user. The microphone and the projector are connected to the mobile phone device. The
projector, projects the information enabling surfaces, walls and physical objects around the
user to be used as interfaces. The microphone is trained frequently with commands; the device
stores these commands which were trained by the user and executes the corresponding action
through the projector.
The user gives the input to the device and when the input is received to the microphone,
the command is processed and is sent to the mobile device .The mobile device interprets the
command and is passed on to the projector. The projector output is seen on the screen through
the mirror for accurate projection. The tips of the user’s fingers are covered with red, yellow,
green and blue colored tapes. Therefore it helps the camera to recognize the hand gestures. The
movements of the user’s fingers recognized into gestures that acts as an interaction instruction
for the device. The Sixth Sense technology is a blend of computer along with a cell phone.
CHAPTER 4
CONCLUSION AND FUTURE SCOPE
4.1 Conclusion
The key here is that Sixth Sense recognizes the objects around you, displaying
information automatically and letting you access it in any way you want, in the simplest way
possible.
Clearly, this has the potential of becoming the ultimate "transparent" user interface for
accessing information about everything around us. If they can get rid of the colored finger caps
and it ever goes beyond the initial development phase, that is. But as it is now, it may change
the way we interact with the real world and truly give everyone complete awareness of the
environment around us.

4.2 Future Scope

Further development in this technology will lead to evolvement of new markets.
Hardware used in the current technology can be perfected as it plays an important role in this
technology by interacting with the user. Security of the current technology can further be
improved and more accuracy should be aimed for. Enhancements should be made so that
visually impaired people can use this technology. The current technology is a little bulky to
carry around, so few improvements can make it user friendly.
 REFERENCES
[1]Sixth Sense 2009 < http://en.wikipedia.org/wiki/SixthSense>
[2]Monika Arora, (2012), Basic Principles of Sixth Sense Technology, VSRD-IJCSIT,
Vol. 2 (8), 687-693.
[3]Abhinav Sharma, Mukesh Agarwal, Anima Sharma, Sachin
[4]Gupta, (2013),Sixth Sense Technology, IJRITCC, Vol. 1(4), 277-282.
[5]S. Sadhana Rao, (2010),Sixth Sense Technology, Proceedings of the International
Conference on Communication and Computational Intelligence, 336-339.
[6]Nalen Anand, Neha Gaur, Kavita Choudhary, (2013), Sixth Sense in a State of
Radical Emergence International Journal of Information and Computation
Technology,Vol. 3,527-532.
[7]Sixth Sense Technology 2011< http://www.slideshare.net/ Sanjay Medichetty/sixth-
sense-technology>
[8]Sixth Sense Technology< http://www.engineersgarage.com/ articles/sixth-sense-
technology >