EYE BLINK TO VOICE AND HOME

AUTOMATION:ASSISTING SYSTEM FOR

PARALYZED
 Vidyarani.H.J
Sahana Sagar H.S Apeksha.B
Assistant Professor
Student Student
Information Science and Engineering
Information Science and Engineering Information Science and Engineering
Dr.Ambedkar Institute Of Technology
Dr.Ambedkar Institute Of Technology Dr.Ambedkar Institute Of Technology
(VTU Affiliation) (VTU Affiliation)
(VTU Affiliation)
Bangalore,India Bangalore,India
Bangalore,India
sahanasagar02@gmail.com apekshabv@gmail.com
vidyaranihj@gmail.com
Sowmya Shree.S
Kavya.V
tudent
Student
Information Science and Engineering
Information Science and Engineering
Dr.Ambedkar Institute Of Technology
Dr.Ambedkar Institute Of Technology
(VTU Affiliation)
(VTU Affiliation)
Bangalore,India
Bangalore,India
sowmyashreeas2126@gmail.com
kavyavdandur@gmail.com

ABSTRACT
This project is a smart
which is specially designed
for MND patients. The
concept of this system is to
apply eye movement to
communicate with the
caretaker and to control the
appliances. This system
comprises of methods like
face detection, eye
detection, eye tracking,
conversion of blink to voice
and blink to home
automation. The image
processing module consist
of webcam and the eye
movement image is
captured and transmitted to
Raspberry Pi
microcontroller for
processing with OpenCV to
derive the coordinate of eye
ball. The system enables
communication using blink
patterns-sequences of long
and short blinks which are
interpreted as semiotic
voice messages.
Keywords—Eye blink
detection, face detection,
eye tracking, conversion of
blink to voice, blink to
home automation,
Raspberry pi
I. INTRODUCTION
Motor Neuron Disease
(MND) is a medical
condition where the motor
neurons of the patient are
paralyzed and it is
incurable. It also leads to
weakness of muscles with
respect to hand, feet or
voice. Because of this, the
patient cannot perform his
voluntary actions and it is
very difficult for the
patients to express his
needs. And even the patient
face major problem like he
won’t be able to
communicate with the
world. A motor nerve gets
damaged and stops working
eventually in MND. Thus,
the nerves that are damaged
connected to the muscles
lose the strength eventually.
There are several subtypes.
In every type, the
symptoms that the patients
face are in different ways.
As the disease promotes,
these symptoms of MND
lean to overlap.
Paralyzed stroke patients
are unable to normally
communicate with their
environment. For these
patients, the only part of
their body that is under
their control, in terms of
muscular movement, is
their eyeballs. Some
research in this area has
focused on investigating
new efficient
communication tools for
paralyzed patients for
translating their eye
movements into appropriate
communication messages.
The biggest problem that
paralyzed patients face
is leading their own life
without anyone else’s help.
This includes basic day to
day operations like
switching on an appliance
or increasing the speed of
fan. Most of the existing
system use sophisticated
hardware and software to
make the control easier and
efficient. But the biggest
problem which a paralyzed
patient faces while using
these system is
accessibility.
The advancement of the
technologies has always
fascinated us. On the other
hand, we also found that,
there are not significant
researches on automation
devices for physically
challenged or disabled
people. Therefore, we
started to look into the
published papers and
innovations around us.
Nowadays medical science
improving day by day. On
this developing procedure
human beings innovating
greater strengthen
scientific accessories such
as smart belt which locate
patient respiration as well
as electro dermal activity
(EDA) sensors to
sequentially display for
physiology symptoms of
seizures at night time.
Medical operations are
now getting easier. Newly input for a microcontroller
developed high-tech in order to control home
gadgets implemented in appliances.
patient’s body to restore
normal activities. In introduction, discussion
Especially paralysis about disease called motor
patients, such as neuron disease is done in
Tetraplegic Patients who which the patient suffers
suffering a lot for their from communication.
physical disabilities. It's There are several
now highly important to techniques introduced for
develop a system which these patients to ease
may help paralysis patients communication. But the
like Tetraplegic Patients. disadvantage is that these
Moreover, people are techniques are costly and
highly interested to digitize uncomfortable. Thus, the
their daily life with less main aim of this project is
physical movement. To to develop an algorithm
fulfill both requirements which is cost effective and
it's high time to develop a user friendly.
system which may help
Tetraplegic Patients as well
as people who are
II. BLOCK
interested to use for
DIAGRAM
efficient and comfortable
life.
Paralysis is one amongst Appliances
Relay
the major neural disorder
that causes loss of motion
of one or more muscles of
the body, wherein
depending on the cause, it
may affect a specific
muscle group or region of
the body, or a larger area
may be involved. In
pursuit of rehabilitation,
the eye can be regarded as
one of the organs that can
Figure.1 Block digram
help a paralyzed person to
communicate suitably. Eye
1.HDMI display screen
movement can be used by
the paralysis patients and
The display screen is used
armless persons to perform
in order to display the eye
simple tasks. This paper
blinking and tracking
describes the acquisition
motion. The screen is
and analysis of eye
mostly important for the
movements for the
caretaker. It can guide him
activation of home
along the proper setup of
appliances for paralysis
the product.
patients. The proposed
method here uses a eye
blink sensor for eye
movement acquisition
thereby reducing the
occurrence of artifacts,
further following a simple
circuitry for
implementation of signal
processing, which is also
cost effective and useful
from the user point of
view. And this processed
signal can be used as an
 connectivity options. The  Image control:
Ethernet in the raspberry Brightness,
pi is the main gateway to contrast,
interconnect with other saturation,
devices and the internet in gamma, white
model B. This draws its balance
power from a micro USB  Image flip:
adapter, with a minimum Horizontal,
range of 2.5 watts(500 vertical
MA). The graphics,  Monitor type:
specialized chip is CRT, LCD
designed to speed up the  Image format:
manipulation of image RGB 24, I420 Figure.5 Raspbian OS logo
calculations. This is in  Power
Figure. 2 HDMI LCD built with Broadcom video consumption: 5. OPEN CV
Display Screen core IV cable, that is 160mW typical OpenCV
useful if you want to run a  Operating abbreviated as open source
For the proper display, we game and video through system: computer version is a
have chosen to use HDMI your raspberry pi. Windows library with functions that
LCD display screen. The 98/2000/ME/X mainly aim real-time
specification of it is shown P/Vista/WIN 7 computer vision. With
below, 3.HD night vision camera
OpenCV one can perform
4.Raspbian face detection using pre-
● 800 x 480 HD trained deep learning face
resolution Raspbian is a detection model which is
● Capacitive touch Debian-based computer shipped with the library
control operating OpenCV is written in C+
● Support Pi system for Raspberry Pi. + and its primary interface
Raspberry There are several versions is in C++, but it still retains
● Support OS: Pi of Raspbian including a less comprehensive
Banana, Pro Raspbian Stretch and though extensive older C
Banana, provide Raspbian Jessie. Since interface.
Ubuntu, Raspbian 2015 it has been officially OpenCV application areas
corresponding provided by the Raspberry include:
mirror Pi Foundation as the
Figure.4 HD
primary operating system  2D and 3D feature
● Support Black BB, night vision
for the family of Raspberry toolkits
provide the camera
corresponding Pi single-board  Egomotion estimat
mirror image computers. Raspbian was ion
It is a plug and play
Angstrom created by Mike Thompson
setup which is easy to
and Peter Green as an  Facial recognition
● General HDMI apply. You can easily
independent project. The system
display, can be make video calls on
used as a computer initial build was completed  Gesture
major IMs. This Intex
monitor in June 2012.The operating recognition
picture motions in 16
● HDMI interface is system is still under active
mega pixels  Human–computer
used to display development. Raspbian is
interpolated digital interaction (HCI)
and USB interface highly optimized for the
clarity, making it apt
is used to touch Raspberry Pi line's low-  Mobile robotics
for both personal as
performance ARM CPUs.
well as professional  Motion
2.Raspberry PI use. It offers high understanding
resolution clear
picture, which is
Fig.3 .Raspberry
suitable for video
pi2 model B
conference.
The Raspberry Pi is a Features:
Broadcom BCM2835  Image
SOC (system on chip resolution: 16.0
board). It comes equipped mega pixels  Object
with a 700 MHz, 512 MB (4608x3456) identification
of SDRAM and interpolated  Segmentation and
ARM1176JZF-S core  Frame rate: Up recognition
CPU. The USB 2.0 port of to 30 fps
the raspberry pi boars uses  Stereopsis stereo
only external data vision: depth
 perception from 2 least supervised life. But performance, considering design basis and verify
cameras the use of prosthetic limbs special needs of a disabled effectiveness of our device.
and artificial organs are still or handicapped people who
 Structure from
not commercially available cannot communicate with 3 Study and development
motion (SFM)
in most of the countries. the outside world using of support tool with blinks
 Motion tracking Recently developed some voice, motion or other for physically handicapped
other brainwave based methods. With the help of children
 Augmented reality
systems are mostly cognitive command, the In this study, we try
expensive, immobile or `Brain-Drive' can be used to develop a new
task specific. In order to to control wheelchair, home application for physically
LITERATURE SURVEY overcome these problems, appliances and even handicapped children to
analogous to the pen-drive, industrial. communicate with others
There are several medical
we have focused on by a blink. Due to limited
disorders that can lead to an
developing a low cost 2 Efficient eye blink body movements and
individual becoming
control unit namely “Brain- detection method for mental disorders, most of
paralyzed or having motor
drive” which transforms disabled helping domain them cannot communicate
speech disorders that
ones thoughts into output Facial paralysis with their families or
inhibits speech or voice
digital electric signal. makes patients lose their caretakers. Think if they
production. Conditions
Certain thoughts (mental facial movements, which can use application in smart
such as Locked-In
tasks) with hard eye can incur eye damage even phones or any other android
Syndrome (LIS) or motor
blinking (neural driven blindness since patients are devices by a blink, it will
neuron diseases such as
physical outcome) develops incapable of blinking. We be big help for them to tell
Amyotrophic Lateral
certain action potential, no design and implement a caregivers what they really
Sclerosis (ALS) and
matter the eye blinking is pair of smart glasses iBlink need or want to tell and
Cerebral Palsy are among
visible or not, it creates a to assist facial paralysis communicate through voice
the common diseases that
significant change in patients to blink. The basic and also control the devices
affect speech. In all or most
electroencephalogram idea is to monitor the like fan, light, or any other
such cases, the patient loses
(EEG) or brain signal. normal side of the face with android or Bluetooth
the ability to communicate
Single channel EEG signal a camera and stimulate the connected devices . Here
with the external world in
is collected from frontal paralysed side, so that the we try to detect an eye area
an effective manner even
lobe as it plays a vital role blink of the both eyes by using Open Cv. Then we
though his intelligence is
in voluntary movement become symmetric. Our develop the way to detect
mostly unaffected. Not only
using a wearable Mind contributions are: First, we opening and closing of
does that cause extreme
wave Mobile from propose an eye-blink eyes. We combine the
distress to that individual,
Neurosky. The EEG data detection mechanism based method using situation and
but also to his family and
thus recorded is on support vector machine using complexity of image
friends. Some customized
preprocessed to reduce the (SVM), which can detect to get more accurate results
Augmentative and
effect of noise and artifacts asymmetric blinks of to detect a blink. The level
Alternative Communication
and then analyzed in time patients under various of handicapped is very
(AAC) devices have been
domain. A grid of cells are illumination conditions varied in children. So we
developed that uses signals
displayed on a monitor in with an accuracy above will try to develop the
from the patient and
front of the subject, which 99%. Our eye-image library application to be able to
converts them into some
is connected to individual for training the model is customize depends on the
form of data that can be
digital output pin onboard published online for further situation of users. And also,
communicated but such
an embedded system. To related studies, which we will try to reduce the
devices are very expensive
select a cell, a user just contains more error to detect a blink and
and are practically out of
needs to look and generate than 30,000 eye images. pursue the high precision of
reach for most people. In
a sustainable eye blink Second, we design and the eye chased program.
this system, we have
which can easily be implement an automatic
designed an extremely low III. METHDOLOGY
identified from the raw stimulation circuits to
priced device that reads and
value of brainwave. generate electrical impulse The methodology used in
converts eye-blinks from
Since eye blinking can be for stimulating the patient's our system is simple and
the patient to a universally
both conscious and facial nerve branches, effective. Use of various
accepted communication
involuntary, automatic which can configure library packages has been
code-The Morse code.
periodic eye blink of very operational parameters in a the main part. As the video
low frequency oscillation is self-adaptive manner for from the camera starts,
1 Brain-drive: a smart
removed to avoid unwanted different patients. Third, we leading to the capturing of
driver for controlling
selection of command. implement the entire iBlink live images. The facial
digital appliances using
Unlike webcam based eye system, which integrates recognition starts with
cognitive command
blink detector, Brain-Drive the two functions above facial landmarking. This is
For the disabled
captures the signal for blink and a communication basically done with the
and handicapped, the world
rather than the visual output function module for tele- help of NumPy library and
of brain-computer interface
of blinking. The system can medicine applications. We Dlib library. NumPy is a
has opened a new horizon
be trained for person conduct experiments in a package in python used in
to lead an unassisted and
specific customized hospital to obtain the faster complex
 mathematical computing
and Dlib is a special kit
containing machine
learning algorithms. With
the help of the these
libraries we are able to
localize and represent
salient features of the face
such as eyes, eyebrows,
nose, mouth and jawline.
These are basically marked
with 68 x-y points and then
finally region of interest is
taken.
No.of blinks Output
1 Fan On
2 Light On B.Detection –
3 Fan Off
4 Light Off
No.of blinks Output
1 Water
2 Food
3 Medicine
4 Alert
The formula for the EAR
(Eye Aspect Ratio) is
given below:
EAR = [(p2-p6) +
(p3-p5)] / [2*(p1-p4)]
Therefore, whenever eye is
closed the points p2 & p6
coincides and points p3 &
p5 coincides and making
the numerator zero. So the
eye aspect ratio becomes
zero which means eye
closed. If the above
mentioned points i.e. p2
with p6 and p3 with p5
does not coincides, it
makes the numerator a
non-zero value so that
means the eye is opened.
Accordingly, the winking
of eye is further detected
and sequences are
generated. So total of 8
sequences in total can be
generated but we have
targeted of only 4 sequence
generation with the
following speech output
giving the basic demands.
Table 1 shown below
shows the sequences and
its output.
The system comprises of 3 the previously developed
phases: prototypes in this field
A.Capturing – because none of the
components are in direct
IV. The image of the driver contact with the patient’s
is captured using HD night body hence it definitely
vision camera, which is will prove to be safer. Use
known for its clarity and of Raspberry pi is simple
cost effective. and also developing
tremendously in the market
V. This camera creates a today. The tool had
video clip and concentrates
advantages over the older
on single frame containing conventional tools.
paralyzed patient eye blink.  To make cost
effective: The main
VI. The captured video is
then divided into frames for
objective of
developing algorithm
analaysing. of a real time video
Oculography system is
that to provide cost
VII. This phase first effective for those
involves the detection people who cannot
of face of the driver. afford. The existing
VIII. Face detection is technique for such
done using facial patients to
landmark which results communicate is too
in locating the face in a costly.
frame. Thus, it is necessary to
IX. Only facial related design a system which
structures or features is affordable to
are detected and all common people which
other types of objects includes cost effective
like buildings, trees, components for
bodies are ignored. designing.
X. In our method eye
 Electrode less
is the decision system: To develop a
parameter for finding system in which the
the state of the driver. patient can
XI. Eye Aspect
communicate without
Ratio(EAR) is the ratio any application of
of number of eye blinks
electrodes. Because
to the width of the eye. this electrodes need to
C.Correction –
be pierced to the skin
 The actual state of the of human body which
eye is found, if it is is very painful. The
closed or open or semi use of electrodes is the
closed or semi open. technique available as
 The identification of of now which is cost
eye status is most effective but it is
important painful and makes the
requirement. patient conscious
 If the systems detects every time and this
that the eyes are open technique is
then it is repeated uncomfortable too.
again and again until  Fast: There are few
closed eyes are found. algorithms which are
developed for video
RESULTS Oculography system
for communication.
The proposed project aims The main objective of
to bring out a solution for this project is to
the paralyzed people develop an algorithm
without any harm to their which is extremely
body externally or
internally. It overweighs
fast compared to the
existing ones.
 Accuracy: The main
objective of this
project is to develop
an algorithm which is
more accurate
compared to the
existing ones.
FUTURE SCOPE
As far as the future of
this system is concerned,
the microcontroller will be
interfaced with an LCD
module to display the
speech output as text
simultaneously. In case, the
caretaker misses out on
listening to the speech
output, he can just see it on
the display and assist the
patient. Further on, to avoid
the clause of dependency of
the patient, Blinkom can be
developed into a human-
machine interface where
different devices can be
interfaced and the
microcontroller can be
programmed in a way that
based on the eye blinks, the
corresponding operation is
also performed.
For example, if the patient
blinks the right eye once
followed by the left eye
once, along with the
corresponding text display
and speech output as “light
on”, the light in the room is
switched on. In this
manner, the patient will
have control over basic
electronic devices around
him. With feature of IoT
coming into picture, we can
handle the needs of the
MND patients from any
place. It is as simple as this
that the demand signal
generated can be passed on
automatically through an
alert signal to the caretaker
and he can with aid of IoT,
can do the function such
switching of light-on or
off / fan-on or off / TV &
AC on or off etc.
CONCLUSION
The proposed project aims
to bring out a solution for
the paralyzed people [3] Masaru Kiyama, Hitoshi using Fn-M16p module”,
without any harm to their Iyatomi and Koichi Ogawa, 2016
body externally or “Development of robust
video-oculography system for
internally. It overweighs
non-invasive automatic nerve
the previously developed quantification”, IEEE
prototypes in this field Conference Publications,
because none of the 2012 IEEE-EMBS
components are in direct Conference on Biomedical
contact with the patient’s Engineering and Sciences.
body hence it definitely
will prove to be safer. Use [4] S. M. H. Jansen, H.
of Raspberry pi is simple Kingma, R. L. M. Peeters and
R. L. Westra, “A torsional eye
and also developing
movement calculation
tremendously in the market algorithm for low contrast
today. The tool had images in video-
advantages over the older oculography”, IEEE
conventional tools. Conference Publications,
As already discussed 2010 Annual International
previously, our device is Conference of the IEEE
very unique when Engineering in Medicine and
compared to the Biology.
conventional devices [5] Akshat Gambhir, K.S
already existing in the Boppaiah, M Shruthi
market. There is no need of Subbaiah, Pooja M and
piercing any electrodes KiranP, RNSIT, “Video
through the epidemics of Oculographic System using
the patient’s skin, hence Real-Time Video Processing”,
very user friendly. Also, by International Journal of
the use of the camera stand Computer Applications (0975
–8887) Volume 119 –No.22,
in our prototype, we are
June 2015.
able to achieve flexibility
by which the device can be [6] Karthik K.P and
set up at any time and at Basavaraju S, Department of
any place without much Electronics & Communication
inconvenience. But,it is the Engineering, Sapthagiri
family’s or the care taker’s College of Engineering,
responsibility to train the Bangalore, “Design and
MND/ paralyzed patient Implementation of Unique
Video Oculographic
regarding the different
Algorithm Using Real Time
combination of eye blinks Video Processing on FPGA”,
required for different type IJSRD -International Journal
of outputs at the speaker. for Scientific Research &
Development| Vol. 3, Issue 04,
2015 | ISSN (online): 2321-
Re 0613.
ferences
[7] Sowmya K S, Roopa B S
and Rishika Prakash, Don
[1] Joshua D. Fischer, Dawie
Bosco Institute of Technology,
J. van den Heever, “Portable
Electronics &
Video-Oculography Device
Communication, Bangalore,
for Implementation in
“A Novel Eye Movement
Sideline Concussion
Recognition Algorithm using
Assessments: A Prototype”,
Video Oculography”, Proc. of
2016 IEEE conference.
Int. Conf. on Recent Trends in
Signal Processing, Image
[2] Sudhir Rao Rupanagudi,
Processing and VLSI,
Vikas N S, Vivek C
ICrtSIV.
Bharadwaj, Manju; Dhruva N;
Sowmya K. S, “Novel
[8] Jacek Naruniec, Michael
methodology for blink
Wiezczorek, Marek Kowalsak
recognition using video
“Webcam-based system for
oculography for
video- oculography”, 2017
communicating”, IEEE
[9] Rachitha S, Prashanth
Conference Publications,
Kumar K.N, “Eyelid
2014, ICAEE.
movement communication
media for paralyzed patient