Eye-Phone

Content

Technology introduction
HPI Evolution
Terminologies
Paper Abstract
Eye Phone Design
Evaluation
Application
Demo
Future work
Improvement on Paper
Related work

Technology Introduction

Human Computer Interaction(HCI)

HCI (human-computer interaction) is the study of how people
interact with computers and to what extent computers are or are
not developed for successful interaction with human beings

Most HCI technology addresses the

interaction between people and computers
in ideal environments, i.e., where people
sit in front of a desktop machine with
specialized sensors and cameras centered
on them

Technology Introduction

Human Phone Interaction(HI)

Human-Computer Interaction (HCI) researchers and phone vendors are

continuously searching for new approaches to reduce the effort users exert
when accessing applications on limited form factor devices such as mobile
phones.
Human-phone interaction (HPI) extends the challenges not typically found in
HCI research, more specially related to the phone and how we use it. In
order to address these goals HPI technology should be less intrusive; that is,
i)
it should not rely on any external devices other than the mobile phone itself;
ii)
it should be readily usable with minimum user dependency as possible;
iii) it should be fast in the inference phase;
iv) it should be lightweight in terms of computation;
v)
it should preserve the phone user experience, e.g., it should not deplete the
phone battery over normal operations.

HPI Evolution - Keypad

1983

HPI Evolution Touch

Screen

1993

HPI Evolution Trackball

2006

HPI Evolution Voice

Operated

2010

HPI Evolution What Next ?

Eye Phone Controlling Phone by Eyes

Terminologies

Intrusive Method:

which requires direct contact with the eyes

Non- Intrusive methods:

which avoid any physical contact with the

user
Model

based method
Appearance based method
Feature based method

Paper Abstract
EyePhone tracks the users eye movement
across the phones display using the
camera mounted on the front of the phone
i) Track the eye and infer its position on the
mobile phone display as a user views a
particular application
ii) Detect eye blinks that emulate mouse
clicks to activate the target application
under view.

EyePhone Design
1)
2)
3)
4)

An eye detection phase

An open eye template creation phase
An eye tracking phase
A blink detection phase

Eye detection phase

By applying a motion analysis technique which operates on

consecutive frames, this phase consists on finding the
contour of the eyes. The eye pair is identified by the left
and right eye contours.

Results of original algorithm

running on a desk-top with a
USB camera
Ref: Eye Tracking and Blink
Detection Library.
(http://tinyurl.com/yb9v4f2)

Results of EyePhone
running on a Nokia N810.
The smaller dots are
erroneously interpreted as
eye contours

Eye detection
Based on previous experimental observations, they modify
the original algorithm
i)

ii)

Reducing the image resolution, which reduces the eye

detection error rate
Adding two more criteria to the original heuristics that filter
out the false eye contours
widthmin width widthmax
heightmin height heightmax

widthmin, widthmax, heightmin, heightmax are thresholds, which identify the

possible sizes for a true eye contour, are determined under various
experimental conditions (e.g., bright, dark, moving, not moving) and
with different people. This design approach boosts the eye tracking
accuracy considerably

Open eye template creation

EyePhone creates a template of a users open
eye once at the beginning when a person uses
the system for the first time using the eye
detection algorithm. The template is saved in
the persistent memory of the device and
fetched when EyePhone is invoked

Downside of this off-line template creation

approach is that a template created in
certain lighting conditions might not be
perfectly suitable for other environments

Eye tracking
The eye tracking algorithm is based on template
matching. The template matching function
calculates a correlation score between the open
eye template, created the first time the
application is used, and a search window.

Outer box around the left eye encloses the region where the
correlation score is calculated. If the normalized correlation
coefficient equals 0.4 we conclude that there is an eye in the
search window. This threshold has been verified accurate by means
of multiple experiments under different conditions (e.g., bright,
dark, moving, not moving)

Eye tracking
Correlation:
1.

2.

Correlation is a measure of the degree to which two variables

agree, not necessary in actual value but in general behavior.
The two variables are the corresponding pixel values in two
images,
template and source.
x is the template gray level image
N 1

( xi x ) yi y

x is the average grey level in the

N 1
N 1
2
xi x yi y 2 template image

y is the source image section

i 0
i 0
y is the average grey level in the
source image
N is the number of pixels in the
section image
(N= template image size = columns *
The value cor is between 1 and +1, with larger values
rows)
cor

i 0

representing a stronger relationship between the two images

Blink Detection
To detect blinks they apply a thresholding technique
for the normalized correlation coefficient returned
by the template matching function
Problem: Quality of the mobile camera is not the same as a
good USB camera, and the phones camera is generally closer
to the persons face than is the case of using a desktop and
USB camera. Because of this latter situation the camera can
pick up iris movements, i.e., the interior of the eye, due to
eyeball rotation. In particular, when the iris is turned towards
the corner of the eye, upwards or downwards, a blink is
weeye
indicate
with open.
cmin and
cmaxoccurs
, respectively,
the
Four
inferred even if Ifthe
remains
This
because
in
thresholds:
minimum
maximum
normalized
this case the majority
of and
the eye
ball surface
turnscorrelation
white
min
T1 which
= 0.64
coefficient
by the template
is confused
with thevalues
color returned
of the skin
T1max =0.75
T2min =-0.53
T2max =-0.45

matching function,
the eye is inferred to be closed if
T1min cmax T1max and T2min cmin T2max
It is inferred to be open otherwise.

EVALUATION
DS = eye tracking accuracy measured in daylight exposure and
being steady;
AS =eye tracking accuracy measured in artificial light exposure and
being steady;
DM = eye tracking accuracy measured in daylight exposure and
walking;
BD = blink detection accuracy in daylight exposure

Device Used: Nokia N810, 400MHz processor, 128 MB of RAM,

Maemo 4.1 OS, OpenCV (Open Source Computer Vision) library,
algorithms compiled on the Maemo scratchbox, video frames
from the camera using GStreamer

EVALUATION
Average CPU usage, RAM usage, and computation time for one video
frame. The front camera supports up to 15 frames per second.
The last column reports the percentage of used battery by
EyePhone after a three hour run of the system

Applications

EyeMenu

Car Driver Safety

EyePhone could also be used to detect drivers

drowsiness and distraction in cars.

FUTURE WORK

Creation of the open eye template and

the filtering algorithm for wrong eye
contours
Improvement over variations of lighting
conditions or movement of the phone in
a persons hand
Using a learning approach instead of a
fixed thresholding policy

Improvement On The Paper

Very Basic Approach used

High speed and better DSP libraries are
available
Need to implement on I-phone and android
phones
Can be incorporated with face detection
Iris tracking
Improvement for various light condition and
face orientation
Online template creation

RELATED WORK
There are a number of developments in HCI related to mobile
phones over the last several years
Use of accelerometers on the phone in order to infer gestures
PhonePoint Pen project : phone being transformed into a pen
to write in the air in order to quickly take small notes without
needing to either write on paper or type on a computer. [4]
uWave project exploits a 3-D accelerometer on a Wii remotebased prototype for personalized gesture recognition. [7]
Eye movement has recently been used for activity recognition.
By tracking the eye researchers show that it is possible to
infer the actions of watching a video, browsing the web or
reading notes. [24]
The eyeSight technology exploits the phone camera to control
the mobile phone by simple hand gestures over the camera.
[5]

Thank You

Questions?