An Implementation of an Eye-blink-based Communication Aid for People
with Severe Disabilities
Muchun Su1,2, Chinyen Yeh1, Shihchieh Lin1, Pachun Wang3, Shawmin Hou3
1
Department of Computer Science & Information Engineering, National Central University,
Taiwan, R.O.C.
2
Graduate Institute of Biomedical Engineering, National Central University, Taiwan, R.O.C.
3
Cathay General Hospital, Taiwan, R.O.C.
E-mail: muchun@csie.ncu.edu.tw
Abstract
This paper presents an implementation of a low-
cost vision-based computer interface which allows
people with severe disabilities to use eye blinks to
access computers and communicate with other
persons. Our communication aid requires only one
low-cost web camera and a personal computer.
Several experiments were conducted to test the
performance of the proposed eye-blink-based
communication aid.
1. Introduction
Computers have been dramatically changing our
lifestyles, livelihoods, and even the whole society.
These kinds of changes benefit some groups in our
society but unavoidably create new barriers to a
disadvantage minority such as people with physical
disabilities who cannot manually access computers
with dexterity as able-bodies people do. Therefore, in
recent years, there has been an effort to design
alternative interfaces for people with disabilities to
replace traditional computer input devices such as
keyboard and mouse.
Assistive technology systems of every variety have
been proposed and even commercialized to allow
people with disabilities to use their limited voluntary
motions to communicate with family and friends,
access computers, and control TVs and air
conditioners, etc [1]-[8]. For some people with severe
disabilities, an extreme disability such as amyotrophic
lateral sclerosis (ALS) or severe cerebral palsy
deprives them of the use of their limbs and even facial
muscles. Owing to this kind of extreme disability,
many available popular assistive technology systems
are not helpful to them. Under this circumstance, eye-
978-1-4244-1724-7/08/$25.00 ©2008 IEEE 351
motion-based systems may provide an alternative
option for people with severe disabilities who only
retain the ability to move their eyes. There are several
different ways to track the eye movements, such as
refection of light [9]-[13] and electrooculographic
potential (EOG) [10], [14]-[19], etc.
Among so many useful assistive technology
systems, the “camera mouse” system [20] and the
“Blink Link” [21] deserve to be particularly
mentioned. The camera mouse system tracks some
small section of a user’s facial features (e.g., nose, lip,
and the whole eye, etc) with a video camera and
translates them into the movements of the mouse
pointer on the screen. By dwelling in the desired screen
area for a certain amount of time, the user may make a
selection or issue a mouse click. The experiences with
the camera mouse system were very encouraging. They
showed that the system could successfully provide
computer access for people with severe disabilities.
However, the eye feature has not been used effectively
with the camera mouse system in their reported work
at that time. Grauman et al. proposed the Blink Link
system which enables communication using eye blink
patterns to provide an alternate input modality to allow
people with severe disabilities to access a computer
[21]. A very high success rate in almost real-time was
reported; however, the system was imposed by some
restrictions. Once the open eye template becomes out
of date for some reasons, the system may give faulty
outputs. In [22], Bhaskar et al. even pointed out that
the Blink Link system suffers from several
disadvantages. For example, it requires offline training
for different depths from the camera for the
computation of the distance. Furthermore, changing
camera positions requires the whole system to be re-
trained.
In our previous work, a vision-based “Head Mouse”
system [23] and an “eye mouse” [24] have been
ICALIP2008
proposed to allow people with disabilities to use their
head movements or eye movements to manipulate
computers. The goal of this paper is to present an
implementation of a non-instructive eye-blink-based
communication aid for the severe disabilities such as
ALS. With the communication aid, ALS people are
able to use their limited voluntary motions such as eye
blinks for communications, manipulating computers,
and controlling home appliances (e.g., TV and air
conditioner, etc). In this communication aid, the Fig. 1 The first layer of the communication
pattern matching technique and optical flow are aid in the English version
integrated to detect eye blinks. The remaining of this
paper is organized as follows. In Section 2, the system
design will be described. Section 3 introduces the
experimental results. Finally, Section 4 concludes the
paper.

2. The proposed communication aid

Our proposed communication aid for people with
severe disabilities dichotomizes daily living necessities
into 7 groups (e.g., Voiced Messages, Typing, Home
Appliance Control, Help, A/V Entertainments, Web
Surfing, Messages) as shown in Fig. 1. In addition to
the seven selections, another two selections, Suspend
and Exit, are another two available options. Most of
the selections’ functionalities are self-evident. For
example, the A/V entertainments selection allows the
user to choose either listening music or watching
movies. By adding more subsequent selections at the
Fig. 2 The “A/V Entertainments” selection
deeper layers as shown in Fig. 2, the user can easily
and its subsequent selections
adjust the volume or switch to another song or movie.
As for the “Voiced Messages” selection, some voiced
messages in common use can be issued by the use via 2.1 Hardware
subsequently blinking the wanted selections. An
example is shown in Fig. 3. Once the user The system consists of a 3.2GHz Pentium 4 PC with
subsequently select the four selections, Body, Head, the Windows XP operating system and a low-cost R-1
Scalp, and Itch, the aid will automatically output the battle snake-one web cam. The Web camera is placed
voice signal “My scalp itches”. The contents of the in front of the computer monitor. The camera supplies
“Voiced Messages” selection are edited and organized 15 color images of size 640 × 480 per second. To
according to the suggestions of “Taiwan Motor Neuron achieve real-time performance, the eye blink detection
Disease Association”. This kind of communications algorithm processes only 320×240 pixels in gray level
provides the ALS patient with the opportunity of at an average 30 frames per second.
expressing movements that he or she wants to do,
feelings, uncomfortableness in body, etc. 2.2 The eye blink detection algorithm
The communication aid sequentially scans through
these nine selections on the row by row basis. The user The applications of eye blink detection and analysis
blinks when his or her desired row is highlighted in red are widely varied from communication aids for the
color. Then the aid scans through each selection in that disabled, driver drowsiness detection, cognitive
row and waits for the blink signal issued by the user. If engagement, operator attentiveness monitoring, etc
there is no detected blink in two complete scans at the [25]-[33]. Among so many different approaches, the
present layer then the aid will automatically jump back frame differencing technique is the most popular
to the upper layer and start to scan the selections at the method used for eye blink detection [21]. Some
upper layer.

352

Fig. 3 The “voice messages” selection and
its subsequent selections
Fig. 4 The four small templates around the
pupil
approaches adopt optical flow [32]-[33]. Each
approach has its own considerations, limitations, and
advantages. Some approaches could achieve a very
high successful rate such as in [21] but some
approaches could only reported 65% success rate such
as in [32]. Of courses, the price paid for achieving a
high success rate may be restrictions in operational
environments. For example, the Blink Link system
could detect eye blinks with a success rate of 96.5% in
almost real-time; however, it suffers from several
restrictions as mentioned in the previous section. In
this paper, we integrate the pattern matching technique
and optical flow to detect eye blinks. It involves in four
steps.
Step 1. Initial eye region location
The system asks the use to blink his or her eye when
the user tries to start to use the communication aid so
that the eye region could be effectively located by the
simple frame differencing method.
Step 2. Templates generating
After the initial eye region has been located, a circle
with an appropriate size will be automatically moved
to encircle the pupil. Then four small rectangle-shaped
templates around the pupil are cropped out of the
353
present image frame and used as the first four
“templates” (as shown in Fig. 4) to determine the
possible position of the eye region in the next image
frame. Since the illumination condition and the
distance between the camera and the user may vary
from time to time during the operation procedure, the
templates should be updated in accordance with the
environmental changes. Otherwise, the templates may
become out of date and wrong eye regions may be
located.
In fact, an intuitive and simple approach for
generating the template for the eye region is to use a
box around the center of the working eye region.
However, from many experimental results, we found
that the performance of the simple template was not as
high as we expected. One possible reason is that the
updated template may gradually lose the representative
of the pupil and then detect wrong regions. That is why
we adopted the four templates since good results could
be expected.
Step 3. Eye tracking
These four templates generated in the previous step
are used for eye tracking. Since we hope the response
time can be minimized we use the pattern matching
technique to locate the eye regions in the subsequent
frames. We use the templates to search for the eye
region in the subsequent frame. For each template, the
template is matched with the test sub-images. Let
t (m, n) and s (m, n) represent the template and the test
sub-image in the current frame, respectively. Template
matching is performed using the following correlation
coefficient
∑ ∑ (t (m, n) − t m )( s(m, n) − s m )
m n
γ = (1)
( ∑∑ t (m, n) − t m )( ∑ ∑ s ( m, n ) − s m )
m n m n
where t m and s m are the averages of the t (m, n) and
s (m, n) . A higher correlation coefficient γ indicates
the two images have similar brightness patterns. The
template with the highest correlation coefficient among
four templates is chosen to be the candidate of the eye
region. If the highest correlation coefficient is larger
than a pre-specified threshold, θ cc , then an eye region
is claimed to be detected and four templates around the
pupil are generated for the next frame. Otherwise, go to
next step and use optical flow to detect whether the
user blinks.
Step 4. Blink detection
Optical flow computation is based on two
assumptions: (1) the brightness of any feature point is
constant over time and (2) nearby points in the image
move in a similar manner. A pyramidal
implementation of a hierarchical optical flow method
[34]-[35] is used to automatically track the 25 anchor
points uniformly distributed in each rectangle-shaped
template. If the average moving length of the 25
anchor points is larger than a threshold, θ of , then a
blink motion is claimed to be detected. One complete
eye blink involves two motions: open-closed followed
closed-open.
An example of the average motion length across
time is depicted in Fig. 5 where two consecutive peaks
represent an eye blink. Moreover, the number of
frames lying between the two peaks may serve as an
indication of a voluntary blink or not. A prolonged
blink with more than three frames between the two
consecutive peaks indicates a voluntary blink. If the
average motion length is less than the threshold, θ of ,
then a blink is not detected. Therefore, we need to go
back to the previous step to search the eye region in a
larger region. If the frequency of miss-detected of the
eye region is higher than a threshold, θ md , then the
system will automatically go to the first step to ask the
user to voluntarily blink to generate another four new
templates.
voluntary blink voluntary blink involuntary blink
Fig. 5 An example of the average motion
length across time and two consecutive peaks
indicate a complete eye blink
3. Experimental results
3.1 Eye blink detection test
The experiment was conducted to test whether the
eye blink detection algorithm can successfully detect
eye blinks under different conditions. Four subjects
were asked to blink and each one was recorded in two
different lighting conditions. We collected a data set
consisting of 8 image sequences taken under varying
lighting conditions. The sequences were manually
examined offline to determine when and how many
eye blinks happened. The total numbers of frames and
eye blinks in these testing sequences were 560 and 40,
respectively. The experimental result shows that the
performance rate could reach 97.5% success rate.
3.2 Typing test
The subjects were asked to use a scanning spelling
program to type “ci lab”. The program organizes the
alphabets into 3 groups. Each group contains 6 rows.
In this spelling program, it takes 2 seconds for a scan.
Six strokes require 18 selections. Without any error in
detecting blinks, it totally requires 98 seconds to
complete the typing task. The average typing time
across the subjects was 114.5 seconds. It indicates that
some blinks were miss-detected so the program took
time to jump back to previous layers. The experimental
result shows that the performance rate can reach
94.75% success rate.
4. Conclusions
In this paper, an implementation of a low-cost eye-
blink-based communication aid for ALS patients is
presented. Experimental results show that it can be
used to manipulate the computer for people via
blinking eyes.
5. Acknowledgements
involuntary blink
This paper was partly supported by the 96CGH-
NCU-A3, the National Science Council, Taiwan,
R.O.C, under the NSC-96-2221-E-008-017, the NSC-
96-2752-E-008-002-PAE, the NSC-96-2524-S-008-
002, and the NSC-96-2422-H-008-001.
6. References
[1] W. J. Perkins and B. F. Stenning, “Control units for
operation of computers by severely physically
handicapped persons,” J. Med Eng. Technol., vol. 10,
no. 1, 1986, pp. 21-23.
[2] O. Takami, N. Irie, C. Kang, T. Ishimatsu, and T.
Ochiai, “Computer interface to use head movement for
handicapped people,” in Proc. IEEE TENCON’96,
Digital Signal Processing Applications, vol. 1, 1996,
pp. 468-472.
[3] D. G. Evans, R. Drew, and P. Blenkhorn, ”Controlling
mouse pointer position using an infrared head-operated
joystick,” IEEE Trans. on Rehabilitation Engineering,
vol. 8, no. 1, 2000, pp. 107-117.
[4] Y. L. Chen, F. T. Tang, W. H. Chang, M.K. Wong, Y.
Y. Shih, and T. S. Kuo, “The new design of an
infrared-controlled human-computer interface for the
disabled,” IEEE Trans. on Rehabilitation Engineering,
vol. 7, Dec. 1999, pp. 474-481.
354
[5] R. B. Reilly and M. J. O’Malley, “Adaptive noncontact
gesture-based system for augmentative
communication,” IEEE Trans. on Rehabilitation
Engineering, vol. 7, no. 2, 1999, pp. 174-182.
[6] M. C. Su, W. C. Cheng, P. Z. Chang, L. Z. Chang, Y.
W. Huang, and C. Y. Tew, “A simple and inexpensive
telephone dialing aid for the disabled,” in IEE
Computing & Control Engineering Journal, vol. 11,
no. 2, April 2000, pp. 73-78.
[7] M. C. Su, C. Y. Chen, S. Y. Su, C. H. Chou, H. F.
Hsiu, and Y. C. Wang, “A Portable Communication
Aid for Deaf-Blind People,” in IEE Computing &
Control Engineering Journal, vol. 12, no. 1, February
2001, pp. 37-43.
[8] M. C. Su, Y. H. Lee, C. H. Wu, S. Y. Su, and Y. X.
Zhao, “Two Low-Cost Human Computer Interfaces for
People with Severe Disabilities,” Biomedical
Engineering – Applications, Basis & Communications,
Vol. 16, No. 6, Dec. 25, 2004, pp. 344-349.
[9] Eye-Trace System, Permobil Meditech AB, Timra,
Sweden, http://www.algonet.se/~eyetrace.
[10] L. Young and D. Sheena, “Survey of eye movement
recording methods,” Behav. Res. Meth. Instrum., vol. 7,
no. 5, 1975, pp.397-429.
[11] T. Hutchinson, K. P. White Jr., W. N. Martin, K. C.
Reichert, and L. A. Frey, “Human-computer interaction
using eye-gaze input,” IEEE Trans. Systems, Man,
Cybernetics, vol. 19, no. 6, 1989, pp. 1527-1533.
[12] G. A. Rinard, R. W. Mateson, R. W. Quine, and R. S.
Tegtmeyer, “An infrared system for determining ocular
position,” ISA Trans., vol. 19, no. 4, 1980, pp. 3-6.
[13] C. H. Morimoto, D. Koons, A. Amit, M. Flickner, and
S. Zhai, “Keeping an eye for HCI,” in Proc. XII
Brazilian Symp. Computer Graphics and Image
Processing, 1999, pp. 171-176.
[14] D. Kumar and E. Poole, “Classification of EOG for
human computer interface,” in the Second Joint
EMBS/BMES Conference, vol. 1, Oct. 2002, pp. 23-26.
[15] J. R. LaCourse and F. C. Hludik Jr., “An eye
movement communication-control system for the
disabled,” IEEE Trans. on Biomedical Engineering,
vol. 37, no. 12, 1990, pp.1215-1220.
[16] P. DiMattia, F. X. Curran, and J. Gips, An Eye Control
Teaching Device for Students Without Language
Expressive Capacity: EagleEes, Lampeter, U.K.:
Edwin Mellen, 2001.
[17] G. Norris, E. Wilson, “The eye mouse: an eye
communication device”, IEEE 23rd Northeast
Bioengineering Conference, May 1997, pp. 66-67.
355
[18] Y. Tomita, Y. Igarashi, S. Honda, N. Matsuo, “
Electro-Oculography Mouse for Amyotrophic Lateral
Sclerosis Patients ”, in the 18th Annual International
Conference of the IEEE Engineering in Medicine and
Biology Society, 1996, pp. 1780 -1781.
[19] K. S. Park and K. T. Lee, ”Eye-controlled
human/computer interface using the line-of-sight and
the intentional blink”, Computer Engineering, Vol. 30,
No.3, pp. 463-473, 1996.
[20] M. Betke, J. Gips, and p. Fleming, “The camera mouse:
visual tracking of body feature to provide computer
access for people with severe disabilities,” IEEE Trans.
on Neural Systems and Rehabilitation Engineering,
vol. 10, no. 1, 2002, pp. 1-10.
[21] K. Grauman, M. Betke, J. Gips, and G. R. Bradski,
“Communication via eye blinks- detection and duration
analysis in real time,” Proc. CVPR 2001, 2001, pp. I-
1010-1017.
[22] T. N. Bhaskar, F. T. Keat, S. Ranganath, and Y. V.
Venkatesh, “Blink detection and eye tracking for eye
location,” TENCON 2003, pp. 821-824.
[23] M. C. Su, S. –Y. Su, and G. –D. Chen, “A low cost
vision-based human-computer interface for people with
severe disabilities,” in Biomedical Engineering-
Applications, Basis, & Communications, Vol. 17, No.
6, 2005, pp. 284-292.
[24] M. C. Su, S. –Y. Su, and G. –D. Chen, “A low cost
vision-based human-computer interface for people with
sevee disabilities,” in Biomedical Engineering-
Applications, Basis, & Communications, Vol. 17, No.
6, 2005, pp. 284-292.
[25] T. Brandt, R. Stemmer, and A. Rakotonirainy,
“Affordable visual driver monitoring system for fatigue
and monotony,” in Proc. IEEE lntemational
Conference on Systems, Man and Cybemetics, vol. 7,
2004, pp.6451-6456.
[26] K. Fukuda, J. A. Stern, T. B. Brown, and M. B. Russo,
“Cognition, blinks, eye-movements, and papillary
movements during performance of a running memory
task,” Aviation, Space, and Environmental Medicine,
vol. 76, July 2005, pp. C75–C85.
[27] M. F. Funada, S. P. Ninomija, S. Suzuki, K. Idogawa,
Y. Yam, and H. Ide, “On an image processing of eye
blinking to monitor awakening levels of human
beings,” in Proc. 18th Annual International Conference
of the Engineering in Medicine and Biology Society,
vol. 3, 1996, pp. 966–967.
[28] Q. Ji, Z. Zhu, and P. Lan, “Real-time nonintrusive
monitoring and prediction of driver fatigue,” IEEE
 Transactions on Vehicular Technology, vol. 53, no. 4,
July 2004, pp. 1052–1068.
[29] H. Lim and V. K. Singh, “Design of healthcare system
for disable person using eye blinking,” in Proc. Fourth
Annual ACIS International Conference on Computer
and Information Science, 2005, pp. 551–555.
[30] P. Smith, M. Shah, and N. da Vitoria Lobo,
“Determining driver visual attention with one camera,”
IEEE Transactions on Intelligent Transportation
Systems, vol. 4, no. 4, December 2003, pp. 205–218.
[31] R. Heishman and Z. Duric, “Using image flow to detect
eye blinks in color videos,” IEEE Workshop on
Applications of Computer Vision, 2007.
356
[32] M. J. Black, D. J. Fleet, Y. Yacoob, “A framework for
modeling appearance change in image sequences,”
International Conference on Computer Vision, 1998.
[33] T. N. Bhaskar, F. T. Keat, S. Ranganath, and Y. V.
Venkatesh, “Blink detection and eye tracking for eye
location,” Conference on Convergent Technologies for
Asia-Pacific Region, vol. 2, 2003, pp. 821-824.
[34] B. D. Lucas and T. Kanade. “An investigation of
smoothness constraints for the estimation of
displacement vector fields from image sequences,”
IEEE Trans. on Pattern Analysis and Machine
Intelligence, vol. 8, 1986, pp. 565-593.
[35] J. Y.Bouguet, “Pyramidal Implementation of the Lucas
Kanade Feature Tracker Description of the algorithm,”
Intel Corporation Microprocessor Research Labs.