PROJECT No.

: 23 DESIGN & DEVELOPMENT OF HUMAN COMPUTER INTERFACE BASED CONTROLLERS

A PROJECT REPORT Submitted in partial fulfillment of the requirements for the award of the degree of BACHELOR OF TECHNOLOGY In ELECTRICAL ENGINEERING

Submitted By: ABHISHEK

Guided By: Dr. VINOD KUMAR

DEPARTMENT OF ELECTRICAL ENGINEERING INDIAN INSTITUTE OF TECHNOLOGY ROORKEE ROORKEE 247667 (INDIA) MAY, 2010

DECLARATION BY THE CANDIDATE

I hereby certify that the work which is presented in this project titled Design and Development of Human Computer Interface based Controllers in partial fulfillment for the award of the degree of Bachelor of Technology (Electrical) submitted to the Department of Electrical Engineering, I.I.T. Roorkee is an authentic record of my own work carried out during the period from August 2010 to May 2010 under the guidance of Dr. Vinod Kumar, Department of Electrical Engineering, I.I.T. Roorkee. The matter embodied in this project report to the best of my knowledge has not submitted for the award of any other degree elsewhere.

Date: Place: Roorkee

Abhishek (B.Tech IV Year)

This is to certify that the above statement made by the candidate is correct to the best of my knowledge

Dr. Vinod Kumar Professor & Head Department of Electrical Engineering IIT Roorkee Roorkee-247667

ii

ACKNOWLEDGEMENT
It is my pleasure to take this opportunity to express my heartfelt gratitude to my guide, Dr. Vinod Kumar, Professor, Department of Electrical Engineering, I.I.T. Roorkee, for his valuable guidance and constant encouragement at every stage of this project, sparing his time, sharing knowledge and experience and providing guidance. It would not have been possible for me to complete this project in time without his suggestion, constant inspiration and timely help. I would like to thank Department of Electrical Engineering, I.I.T. Roorkee for resources that helped me to complete our work. I am especially thankful to lab staff for their cooperation. I take this opportunity to thank Mr. Atul Bansal, S.R.F. (Biomedical Laboratory, IIT Roorkee) for his valuable help throughout my project. I express my heartiest gratitude and love to my parents, friends for their patience, encouragement and constant inspiration. Above all I thank God for his blessings.

Abhishek

iii

ABSTRACT
This project is for cursor control in a real world environment, where people with motor disabilities face many challenges to operate a personal computer. The projects main objective is to demonstrate the ability of ocular signals in controlling the cursor of the computer. Ocular signals from FP1 and FP2 positions are used to simulate horizontal cursor movement and cursor clicks. The main challenges of the technology demonstrator are difference in the EEG signal patterns of different people, computational complexity and data acquisition. Signal is acquired from the FP1 and FP2 electrode positions using BIOPAC MP150 and EEG100C systems. Acquisition program was developed in Dev C++ and data analysis program and GUI were developed in MATLAB. Communication between both the compilers was established using text files which act as interrupts for both the programs. A special filter is used to remove the power line frequency and noise. Data analysis is done by finding the differential of the signal using curve fitting or non linear regression technique. Maxima and minima points are calculated from the differential vector and then analyzed in order to reduce the number of maxima and minima so that they represent maximum speed of eyelid closure and opening. The indices of maxima and minima are used to differentiate between an eyeblink or eyeball movement. A GUI was developed which acted as the computer interface with a defined event for getting Visually Evoked Event Related Potentials required by the analysis tools. The event consists of a figure with five rectangular boxes, out of which one is to be selected by moving the cursor over it and then clicking. Cursor can be clicked by the blinking of eyelid while discrete movement of cursor can be done by moving the eyeball in the desired direction. This interface provides a simple method of communicating with the computer for persons with motor disabilities.

iv

Table of Contents
DECLARATION BY THE CANDIDATE ACKOWLEGEMENT ABSTRACT Table of Contents List of Figures Introduction Background Theory 1. Electroencephalography 1.1. Introduction 1.2. Source of EEG Activity 1.3. Methodology 1.3.1 10-20 System (EEG) 1.3.2 Recording EEG Data 1.4. EEG Waves 1.5. Limitations 1.5.1 Artifacts in EEG 2. Non-Linear Regression 2.1. Introduction 2.2. Mathematical Analysis 2.2.1 Non-Linear Least Squares (Theory) 2.2.2 Non-Linear Least Squares (Geometric Implementation) 2.3. Computational Methods 2.3.1 Vanilla Descent Method 2.3.2 Gauss Newton Method 2.3.3 Levenberg Marquardt Algorithm 2.4. An Example 3. EEG Recording System 3.1. Introduction 3.2. MP150 Data Acquisition System 3.3. UIM100C 3.4. EEG100C 3.5. EEG Electrodes 4. EEG Signal Acquisition 4.1. Placement of Electrodes 4.2. Eyeblink 4.3. Eyeball Movement 4.4. Filtering 4.4.1 Low Pass Filter 4.4.2 High Pass Filter 4.5. Algorithm 4.5.1. Main Process 4.5.2. Data Analysis Project Work Stage1: Data Acquisition 1 ii iii iv 1 3 4 6 6 6 6 7 7 8 9 11 11 13 13 13 14 15 16 16 16 17 18 21 21 21 22 23 24 25 25 25 26 28 28 29 30 31 31 34 34

1. MP150acq 2. MATLAB GUI 3. Trial Run Stage2: Eyeblink Detection and Eyeball Tracking 1. List of Main Functions 2. Calculation of Differential Data-avgresults 2.1 Purpose and Prototype 2.2 Run and Demonstration 3. Locating Maxima-locmaxmin 3.1 Purpose and Prototype 3.2 Run and Demonstration 4. Locate Final Peaks-maxminanalyse 4.1 Purpose and Prototype 4.2 Run and Demonstration 5. Sorting time series-sort1 5.1 Purpose and Prototype 5.2 Run and Demonstration 6. trackingeye 6.1 Purpose and Prototype 7. runderiv 7.1 Purpose and Prototype 8. Cursor Initialization and Movement in MATLAB Stage3: Implementation of Graphical User Interface (GUI) 1. Graphical User Interface 2. Description of Different Controls of GUI Practical Application Human Computer Interf ace 1. Introduction 2. Main Features 3. Block diagram of Human Computer Interface 4. Requirements 4.1 Hardware Requirements 4.2 Software Requirements 5. Operating Instructions 5.1 Starting the System 5.2 Normalization 5.3 Running in Real-Time Mode 5.4 Running in Offline Mode 6. The Experiment 6.1 Instructions for Cursor Control Future Work Conclusion Bibliography & References

34 35 35 38 38 38 38 38 39 39 39 40 40 40 41 41 41 42 42 42 42 42 43 43 43 48 48 48 48 49 49 49 49 49 49 50 50 50 50 50 51 52 53

List of Figures
Figure 1: A Typical BCI System Figure 2: International 10-20 System Figure 3: Delta Wave Figure 4: Theta Wave Figure 5: Alpha Wave Figure 6: Mu Rhythm Figure 7: Beta Wave Figure 8: Gamma Wave Figure 9: Initial Condition Figure 10: Condition After 10 Iterations Figure 11: Final Condition Figure 12: MP150 Data Acquisition System Figure 13: UIM 100C Figure 14: EEG 100C Figure 15: Placement of Electrodes Figure 16: Eyeblink Data Figure 17: Movement towards Left Figure 18: Movement towards Right Figure 19: Sequence of Right and Left Movements Figure 20: Result of Filtering Figure 21: Flowcharts Figure 22: Function Fitting Using Curve Fitting Toolbox Figure 23: Block Diagram of Data Acquisition and Communication Figure 24: Output of .exe at start Figure 25: Output after end of acquisition Figure 26: Differential of Data Figure 27: vmaxmin on differential data Figure 28: Erroneous Data Figure 29: Main GUI Window Figure 30: Exit Dialog Box Figure 31: Start Stop Control Figure 32: Menu Items Figure 33: Mode Dialog Box Figure 34: Eyeblink Normalization Figure 35: Eyeball Normalization Figure 36: Experimental Setup 4 8 9 9 10 10 10 11 18 19 19 21 23 23 25 26 27 27 28 29 30 32 35 36 36 39 40 41 43 44 44 45 45 46 47 48

Introduction
Brain Computer Interface is one of the hopeful technologies for communication between humans and machines. Electroencephalogram-based Brain Computer Interfaces have become a major topic of research in the field of neural engineering, rehabilitation, and brain science. Brain computer interface (BCI) [1][2][3] technology has been used as a means for providing individuals with severe motor disabilities, an ability to communicate via computers and to have command and control over their environment. Figure 1 shows a typical setup for BCI. An important issue in BCI research is cursor control, where the objective is to map the EEG signal to movements of a cursor on a computer screen. It has many potential applications, not only in the control of computer cursors, but also in other devices like robotic arms or neuroprostheses. BCI can be divided into two categories i.e. invasive and noninvasive. Most noninvasive BCI systems use electroencephalogram (EEG) signals; i.e., the electrical brain activity recorded from electrodes placed on the scalp. Measuring the EEG is a simple noninvasive way to monitor electrical brain activity, but it does not provide detailed information on the activity of single neurons (a few Volts) and noisy environment (especially if recording outside shield rooms). In invasive BCI systems, the activity of single neurons (their spiking rate) is recorded from microelectrodes implanted in the brain. These invasive BCIs face substantial technical difficulties and entail significant clinical risks as they require that recording electrodes be implanted in the cortex and function well for long periods, and they risk infection and other damage to the brain. For humans, therefore, noninvasive BCI systems are applied due to the clinical risks and ethics.

Fig. 1 A Typical BCI Setup [5]

Level of motor disability in patients largely depends upon the clinical causes of impairment. In the most affected patients, the locked-in group, there is no residual 4

motor ability. As there are no other means available to the patient to communicate with outside world, both invasive and non-invasive BCI can be useful. The relative number of these cases with no residual disability is rare: much more common are cases in which arm function is lost, but facial and eye muscle control remain intact. In such cases, noninvasive means of restoration of reaching and grasping promises to offer significant benefits. Incorporation of new features in BCI is being carried out by researchers all over the world. This project uses the eyeblink detection and eyeball tracking methodologies to establish cursor control in 1 dimension. Eyeblinks and eyeball movement generates a potential in the FP1 and FP2 electrode positions of EEG. This visually evoked potential can be translated into cursor movement and clicks. The human computer interface that has been developed in this project is a non-invasive type of BCI and instead of relying on the thinking process in the brain; it uses ocular muscle movements to guide a cursor.

Background Theory
1. Electroencephalogram
The electroencephalogram (EEG) [9] is a record of the oscillations of brain electric potentials recorded from perhaps 20 to 256 electrodes attached to the human scalp. The recorded signals are transmitted to an EEG system composed of amplifiers, filters, and paper chart or computer monitor. The first human EEG recordings were accomplished by the German psychiatrist Hans Berger in 1924. Bergers subjects (typically his children) revealed robust changes when the eyes were closed and when the subjects performed mental arithmetic.

1.1 Introduction

1.2 Source of EEG Activity

The electrical activity of the brain can be described in spatial scales from the currents within a single dendritic spine to the relatively gross potentials that the EEG records from the scalp. Neurons, or nerve cells, are electrically active cells which are primarily responsible for carrying out the brain's functions. Neurons create action potentials, which are discrete electrical signals that travel down axons and cause the release of chemical neurotransmitters at the synapse, which is an area of near contact between two neurons. This neurotransmitter then activates a receptor in the dendrite or body of the neuron that is on the other side of the synapse, the post-synaptic neuron. The neurotransmitter, when combined with the receptor, typically causes an electrical current within the dendrite or body of the post-synaptic neuron. Thousands of post-synaptic currents from a single neuron's dendrites and body then sum up to cause the neuron to generate an action potential. This neuron then synapses on other neurons, and so on. EEG reflects correlated synaptic activity caused by post-synaptic potentials of cortical neurons. The ionic currents involved in the generation of fast action potentials may not contribute greatly to the averaged field potentials representing the EEG. More specifically, the scalp electrical potentials that produce EEG are generally thought to be caused by the extracellular ionic currents caused by dendritic electrical activity, whereas the fields producing magneto-encephalographic signal are associated with intracellular ionic currents. The electric potentials generated by single neurons are far too small to be picked by EEG. EEG activity therefore always reflects the summation of the synchronous activity of thousands or millions of neurons that have similar spatial orientation, radial to the scalp. Currents that are tangential to the scalp are not picked up by the EEG. The EEG therefore benefits from the parallel, radial arrangement of apical dendrites in the cortex. Because voltage fields fall off with the fourth power of the radius, activity from deep sources is more difficult to detect than currents near the skull. Scalp EEG activity shows oscillations at a variety of frequencies. Several of these oscillations have characteristic frequency ranges, spatial distributions and are associated 6

with different states of brain functioning (e.g., waking and the various sleep stages). These oscillations represent synchronized activity over a network of neurons. The neuronal networks underlying some of these oscillations are understood, while many others are not. Research that measures both EEG and neuron spiking finds the relationship between the two is complex with the power of surface EEG only in two bands that of gamma and delta relating to neuron spike activity.

1.3 Methodology
In conventional scalp EEG, the recording is obtained by placing electrodes on the scalp with a conductive gel or paste, usually after preparing the scalp area by light abrasion to reduce impedance due to dead skin cells. Many systems typically use electrodes, each of which is attached to an individual wire. Some systems use caps or nets into which electrodes are embedded; this is particularly common when high-density arrays of electrodes are needed. Electrode locations and names are specified by the International 1020 system for most clinical and research applications (except when high-density arrays are used). This system ensures that the naming of electrodes is consistent across laboratories. 1.3.1 10-20 system (EEG) The "10-20" system or "International 10-20" system is an internationally recognized method to describe and apply the location of scalp electrodes in the context of an EEG test or experiment. This method was developed to ensure standardized reproducibility so that subjects studies could be compared over time and subjects could be compared to each other. This system is based on the relationship between the location of an electrode and the underlying area of cerebral cortex. The "10" and "20" refer to the fact that the actual distances between adjacent electrodes are either 10% or 20% of the total frontback or right-left distance of the skull. Each site has a letter to identify the lobe and a number to identify the hemisphere location. The letters F, T, C, P and O stand for Frontal, Temporal, Central, Parietal and Occipital respectively. Note that there exists no central lobe; the "C" letter is only used for identification purposes only. A "z" (zero) refers to an electrode placed on the midline. Even numbers (2, 4, 6, 8) refer to electrode positions on the right hemisphere, whereas odd numbers (1, 3, 5, 7) refer to those on the left hemisphere. Two anatomical landmarks are used for the essential positioning of the EEG electrodes: first, the nasion which is the point between the forehead and the nose; second, the inion which is the lowest point of the skull from the back of the head and is normally indicated by a prominent bump. When recording a more detailed EEG with more electrodes, extra electrodes are added utilizing the spaces in-between the existing 10-20 system. This new electrode-namingsystem is more complicated giving rise to the Modified Combinatorial Nomenclature (MCN). This MCN system uses 1, 3, 5, 7, 9 for the left hemisphere which represents 10%, 20%, 30%, 40%, and 50% of the inion-to-nasion distance respectively. The introduction 7

of extra letters allows the naming of extra electrode sites. Note that these new letters do not necessarily refer to an area on the underlying cerebral cortex.

Fig. 2 International 10-20 System (Courtesy: http://frontalcortex.com/)

1.3.2 Recording EEG Data In most clinical applications, 19 recording electrodes (plus ground and system reference) are used. A smaller number of electrodes are typically used when recording EEG from neonates. Additional electrodes can be added to the standard set-up when a clinical or research application demands increased spatial resolution for a particular area of the brain. High-density arrays (typically via cap or net) can contain up to 256 electrodes more-or-less evenly spaced around the scalp. Each electrode is connected to one input of a differential amplifier (one amplifier per pair of electrodes); a common system reference electrode is connected to the other input of each differential amplifier. These amplifiers amplify the voltage between the active electrode and the reference (typically 1,000100,000 times, or 60100 dB of voltage gain). In analog EEG, the signal is then filtered, and the EEG signal is output as the deflection of pens as paper passes underneath. Most EEG systems these days, however, are digital, and the amplified signal is digitized via an analog-to-digital converter, after being passed through an anti-aliasing filter. Analog-to-digital sampling typically occurs at 256512 Hz in clinical scalp EEG; sampling rates of up to 20 kHz are used in some research applications. During the recording, a series of activation procedures may be used. These procedures may induce normal or abnormal EEG activity that might not otherwise be seen. These procedures include hyperventilation, photic stimulation (with a strobe light), eye closure, mental activity, sleep and sleep deprivation. During epilepsy monitoring, a patient's typical seizure medications may be withdrawn. 8

The digital EEG signal is stored electronically and can be filtered for display. Typical settings for the high-pass filter and a low-pass filter are 0.5-1 Hz and 3570 Hz, pass respectively. The high-pass filter typically filters out slow artifact, such as -pass electrogalvanic signals and movement artifact, whereas the low pass filter filters out low-pass high-frequency artifacts, such as electromyographic signals. An additional notch filter is frequency typically used to remove artifact caused by electrical power lines. A typical adult human EEG signal is about 10V to 100 V in amplitude when measured from the scalp and is about 1020 mV when measured from subdural electrodes. 20 Since an EEG voltage signal represents a difference between the voltages at two electrodes, the display of the EEG for the reading encephalographer may be set up in one of several ways. The representation of the EEG channels is referred to as a montage.

1.4 EEG Waves [9]

Following is the description of different types of waves in EEG signal:

Delta is the frequency range up to 4 Hz. It tends to be the highest in amplitude and the slowest waves. It is seen normally in adults in slow wave sleep It is also seen sleep. normally in babies. It may occur focally with subcortical lesions and in general distribution with diffuse lesions, metabolic encephalopathy hydrocephalus or deep midline lesions.

Fig. 3 Delta Wave (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography http://en.wikipedia.org/wiki/Electroencephalography)

Theta is the frequency range from 4 Hz to 7 Hz. Theta is seen normally in young children. It may be seen in drowsiness or arousal in older children and adults; it can also be seen in meditation. Excess theta for age represents abnormal activity. . It can be seen as a focal disturbance in focal subcortical lesions. lesions.

Fig. 4 Theta Wave (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography wikipedia.org/wiki/Electroencephalography)

Alpha is the frequency range from 8 Hz to 12 Hz. Hans Berger named the first rhythmic EEG activity he saw, the "alpha wave." This is activity in the 812 Hz 8 9

range seen in the posterior regions of the head on both sides, being higher in amplitude on the dominant side. It is brought out by closing the eyes and by relaxation.

Fig. 5 Alpha Wave (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography http://en.wikipedia.org/wiki/Electroencephalography)

Mu rhythm is alpha-range activity that is seen over the sensorimotor cortex. It s alpha cortex characteristically attenuates with movement of the contralateral arm (or mental imagery of movement of the contralateral arm).

Fig. 6 Mu Rhythm (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography http://en.wikipedia.org/wiki/Electroencephalography)

Beta is the frequency range from 12 Hz to about 30 Hz. It is seen usually on both sides in symmetrical distribution and is most evident frontally. Beta activity is closely linked to motor behavior and is generally attenuated during active movements. Low amplitude beta with multiple and varying frequencies is often amplitude associated with active, busy or anxious thinking and active concentration.

Fig. 7 Beta Wave (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

Gamma is the frequency range approximately 30100 Hz. Gamma rhythms are 30 100 thought to represent binding of different populations of neurons together into a network for the purpose of carrying out a certain cognitive or motor function. fun 10

Fig. 8 Gamma Wave (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography http://en.wikipedia.org/wiki/Electroencephalography) Table 1 Description of EEG Waves

Type Delta Theta Alpha Beta Gamma

Frequency (Hz) upto 4 4-7 8-12 12-30 30-100+

Location Frontally in adults, posterior in children; high amplitude waves. Found in locations not related to task at hand. Posterior regions of head, both sides, higher in amplitude on dominant side. Central sites (c3-c4) at rest. c4) Both sides, symmetrical distribution, most evident frontally; low amplitude waves. Somatosensory cortex

1.5 Limitations
EEG has several limitations. Most important is its poor spatial resolution. EEG is most sensitive to a particular set of post-synaptic potentials: those which are generated in post synaptic superficial layers of the cortex, on the crests of gyri directly abutting the skull and radial to the skull. Dendrites which are deeper in the cortex, inside sulci, in midline or deep , structures (such as the hippocampus), or producing currents which are tangential to the hippocampus), skull, have far less contribution to the EEG signal. The meninges, cerebrospinal fluid and skull "smear" the EEG signal, obscuring its intracranial source. It is mathematically impossible to reconstruct a unique intracranial current source for a given EEG signal, as some currents produce potentials that cancel each other out. This is referred to as the inverse problem. However, much work has been . done to produce remarkably good estimates of, at least, a localized electric dipole that represents the recorded currents. 1.5.1 Artifacts in EEG Eye-induced artifacts are caused by the potential difference between the cornea and induced retina, which is quite large compared to cerebral potentials. When the eye is completely , still, this does not affect EEG. But there are nearly always small or large reflexive eye movements, which generates a potential which is picked up in the frontopolar and frontal fr leads. Involuntary eye movements, known as saccades, are caused by ocular muscles, , which also generate electromyographic potentials. Purposeful or reflexive eye blinking also generates electromyographic potentials, but more importantly there is reflexive

11

movement of the eyeball during blinking which gives a characteristic artifactual appearance of the EEG. Eyelid fluttering artifacts of a characteristic type were previously called Kappa rhythm (or Kappa waves). It is usually seen in the prefrontal leads, that is, just over the eyes. Sometimes they are seen with mental activity. They are usually in the Theta (47 Hz) or Alpha (813 Hz) range. They were named because they were believed to originate from the brain. Later study revealed they were generated by rapid fluttering of the eyelids, sometimes so minute that it was difficult to see. They are in fact noise in the EEG reading, and should not technically be called a rhythm or wave. Therefore, current usage in electroencephalography refers to the phenomenon as an eyelid fluttering artifact, rather than a Kappa rhythm (or wave).

12

Background Theory
2. Non-linear Regression
2.1 Introduction
Regression analysis is a method used to find the best fit line or curve for the series of data points. It can be used for prediction (including forecasting of time-series data), inference, and hypothesis testing, and modeling of causal relationships. Nonlinear regression [4][5] aims to describe the relationship between a response variable and one or more explanatory variables in a non-linear fashion. Nonlinear regression is more general than linear regression and can fit data to any equation that defines Y as a function of X and one or more parameters. It finds the values of those parameters that generate the curve that comes closest to the data.
Table 2 Comparison between non-linear and linear regression

Non Linear Regression

Dependent variable, yi is not a linear combination of the parameters It is not possible to directly derive an equation to compute the best-fit values from the data, hence the sum of squares function must be minimized by an iterative procedure. Non-convergence may be a problem.

Linear Regression
Dependent variable, yi is a linear combination of the parameters Minimization of squared error results in a set of simultaneous linear equations in the parameters, which are solved to yield the parameter estimators It is globally concave and hence convergent

Uses matrix algebra as basis

Uses simple algebra as basis

2.2 Mathematical Analysis

Nonlinear regression model can be given by a general equation:
, x1 i i

, where the function f() relating the response to the predictors is not necessarily linear. is the vector of parameters. The likelihood for this model is given by: , 1 2 exp 2
, x1 i

13

is minimized. Differentiating S() with respect to parameters we get: 2

, x1 i d , x1 i

Setting the partial derivatives to zero produces estimating equations for the parameters. These equations are generally non-linear and require numerical optimization methods for their solution. Coefficient variances may be estimated from the linearised version of the model. Let,
d , x1 i

, and F={Fij}. Then the estimated asymptotic covariance matrix of the regression parameters is

,where s2 is the estimated error variance. 2.2.1 Non Linear Least Squares (Theory) It is desired to find the vector of parameters such that the curve fits best the given data in the least squares sense, that is, the sum of squares

,where residual zero.

, xi for i = 1.....m. For minimum error gradients should be

1, . . ,

In a non-linear system, the derivatives are functions of both the independent variable and the parameters, so these gradient equations do not have a closed solution. Instead, initial values must be chosen for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation, ~

14

At each iteration, the model is linearized by approximation to a first-order Taylor series expansion about as given by d , k , ~ , j k ~ , The Jacobian, J, is a function of constants, the independent variable and the parameters, so it changes from one iteration to the next. Thus, in terms of the linearized model,

, k

On substitution, gradient equations become: 2 0

which on rearrangement gives the normal simultaneous equations

When the observations are not equally reliable, a weighted sum of squares may be minimized,

Each element of the diagonal weight matrix, W should, ideally, be equal to the reciprocal of the variance of the measurement. The normal equations are then

2.2.2 Non Linear Least Squares (Geometric Interpolation) In Nonlinear Least Squares, the objective function is quadratic with respect to the parameters only in a region close to its minimum value, where the truncated Taylor series is a good approximation to the model. ~

15

The more the parameter values differ from their optimal values, the more the contours deviate from elliptical shape. A consequence of this is that initial parameter estimates should be as close as practicable to their optimal values. It also explains how divergence can come about as the Gauss-Newton algorithm is convergent only when the objective function is approximately quadratic in the parameters.

2.3. Computational Methods

Next task in non-linear regression is to arrive at the solution of the least square equations that we have formed. This task requires iterative techniques so that the squared error is least. There are many ways of solving these equations. 2.3.1 Vanilla Descent Method Simplest, most intuitive technique Parameter update is performed by adding the negative of the scaled gradient at each step

Convergence problems like different curvature at different points and update rule tends to rattle out minima sometimes. We should take large steps where gradient descent is gentle and small steps where gradient descent is more but this method does the opposite.

2.3.2 Gauss Newton Method Quadratic convergence method, and utilizes curvature information is updated as H 1g where H = Hessian Matrix given by

2
g= gradient vector of S()

2
Ignore second order derivative terms. This gives ~2 16

,where This leads to Gauss-Newton method given by:

;
Convergence of the GaussNewton method is not guaranteed in all instances. The approximation:
2

needs to hold to be able to ignore the second-order derivative terms may be valid in the following two cases, for which convergence is to be expected: The function values ri are small in magnitude, at least around the minimum. The functions are only "mildly" non-linear, so that is relatively small in

magnitude 2.3.3 Levenberg Marquardt Method [10] This algorithm is by far the most widely used algorithm as it can change its characteristics according to the type of error. For cases, where vanilla descent is efficient, Levenberg-Marquardt (LM) [10] algorithm approaches vanilla descent and for cases, where gauss-newton method is fast and efficient, LM algorithm approaches the gaussnewton method. LM algorithm was proposed in two steps. Levenberg made an algorithm which was modified by Marquardt and this modified algorithm is known as LM algorithm. Contributions by Levenberg Noticed that GNA and gradient descent are complimentary to each other. Modified the update rule as:

Algorithm to use the update rule: i. ii. iii. iv. Do an update as directed by the rule above. Evaluate the error at the new parameter vector. If the error has increased as a result the update, then retract the step and increase by a factor of 10 or so. Then go to (1) and try an update again. If the error has decreased as a result of the update, then accept the step (i.e. keep the weights at their new values) and decrease l by a factor of 10 or so. 17

There were some limitations in the algorithm proposed by Levenberg. If the value of is large then the Hessian matrix is not used at all. This results in extra calculations for Hessian matrix which becomes irrelevant in this case. It was not able to resolve the classical error valley problem i.e. it didnt result in larger movement at places where error gradient is very small. Contributions by Marquardt Provided insight to solve the classical 'valley' problem Modified the update rule as

Since the Hessian is proportional to the curvature of f, this implies a large step in the direction with low curvature (i.e., an almost flat terrain) and a small step in the direction with high curvature (i.e. a steep incline).

Hence, the limitations of Levenberg algorithm were removed by Marquardt algorithm and this resulted in the formation of LM algorithm which is considered to be the most robust method for non-linear regression.

2.4. An Example [9]

Let us study an example that has been taken from Wikipedia. Original data is provided by the user and now we need to form a model which when used can regenerate the data within the permissible error limits. Let the model be y=acos(bx) + bsin(ax) Initial values of parameters: a=100, b=102

Fig. 9 Initial Condition (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

Green line represents original data and blue line represents the fitted data. With initial values of parameters the plot looks like one shown in Fig. 9. Our aim is to change the 18

value of parameters iteratively to arrive at a point where the squared error is within the tolerance values. At this point the fitted curve would start touching the original data and we can visually observe that both the curves are almost identical to each other. There are various parameters like confidence intervals, chi squared error etc. which can help us analyze the goodness of fit and careful study of these parameters help us in choosing the right model for our data.

Fig. 10 Condition after 10 iterations (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

Fig. 11 Final Condition (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

19

Following are the steps involved in performing non-linear regression for a given data: Clarify your goal, is nonlinear regression appropriate Prepare the data Choose a model Decide which model parameters to fit and which to constrain Choose a weighing scheme (if any) Choose initial values Perform curve-fit and analyze the results.

20

Background Theory
3. EEG Recording System
3.1 Introduction
BIOPAC EEG System [11] is a system that consists of three different components which work in tandem to produce the required EEG signal from the user. This system can be used to acquire, communicate and analyze the data. Three different components are: MP150 UIM100C EEG100C

3.2 MP150 Data Acquisition System

The MP150 Data Acquisition System is a device that acquires data and communicates with the computer by sending the data from the amplifier to the computer for analysis purposes. AcqKnowledge is an interactive, intuitive program that lets the user instantly view, measure analyze and transform data from the MP150 System. Following are the salient features of this device: Ethernet-ready data acquisition and analysis Record multiple channels with differing sample rates Record at speeds up to 400 kHz

Applications of this device include amplifiers and interfaces, biomechanics, ECG, EEG, EMG, EOG, exercise physiology, sleep studies etc. MP150 System components include: Ethernet-ready data acquisition unit Universal Interface Module Network interface card Ethernet crossover cable 12V Power transformer with cord

Fig 12 MP150 Data Acquisition System (Courtesy: http://www.biopac.com)

21

Table 3 Specifications of MP150 System

Analog Input

Analog Output

Digital I/O

Time Base

Number of Channels Absolute Maximum Input Operational Input Voltage A/D Resolution Accuracy (% of FSR) Input impedance Number of Channels Max number of waveforms output with acquisition Output Voltage Range D/A Resolution Accuracy (% of FSR) Output Drive Current Output Impedance Number of Channels Voltage Levels Output Drive Current External Trigger Input Min Sample Rate Trigger Options Amplifier Module Isolation CE Marking Leakage current Fuse Ethernet Power Requirements Transmission Type Analog Output Buffer Internal Buffer

16 15 V 10 V 16 Bits 0.003 1.0 MOhm 2 2 10 V 16 Bits 0.003 5 mA 100 Ohm 16 TTL, CMOS 20 mA (max) TTL, CMOS 2 samples/hour Internal, External or Signal Level Provided by the MP unit EC Low Voltage and EMC Directives <8 A (Normal), <400 A (Single Fault) 2 A (fast blow) UDP (10M bits/sec) 12VDC @ 2 amp Ethernet 512 Ks per analog output channel 6 M samples

Power

Serial Interface

Miscellaneous

3.3 UIM100C
The UIM100C (Universal Interface Module) is used to connect amplifier modules and signal cables to an MP system. Sixteen analog input channels plus 2 analog output channels are provided via standard 3.5 mm phone jacks on the front panel. The back panel provides access to 16 digital I/O lines and an external trigger input via screw terminals. Amplifier modules snap onto the side of the UIM100C for signal routing to the MP data acquisition unit. The output of any amplifier module snapped to the UIM100C can be monitored during acquisition via the corresponding 3.5 mm analog channel phone jack. The UIM100C also provides a direct link to the analog and digital I/O lines of the MP device when collecting or sending data to external equipment. 22

Fig. 13 UIM 100C (Courtesy: http://www.biopac.com) Table 4 Specifications of UIM100C Module

Analog I/O Access D/A Outputs Digital I/O Access Ext. Trigger Access Isolated Power Access

16 channels (front panel) - 3.5 mm phone jacks 2 channels (front panel) - 3.5 mm phone jacks 16 channels (back panel) - screw terminals 1 channel (back panel) - screw terminal 12 V, 5 V @ 100 ma (back panel) - screw terminals

3.4 EEG100C
The EEG100C Electroencephalogram (EEG) amplifier amplifies bioelectric potentials associated with neuronal activity of the brain and can be used to perform unipolar or bipolar EEG measurements. The amplifier output can be switched between normal EEG output and Alpha wave detection. The Alpha detection mode outputs a smoothed wave with a peak indicating maximal alpha activity (signal energy in the 8-13 Hz frequency range).

Fig. 14 EEG 100C Amplifier (Courtesy: http://www.biopac.com) Table 5 EEG 100C Amplifier

Gain Output selection Low Pass Filter High Pass Filter Notch Interference Filter Zin CMRR

5000, 10000, 20000, 50000 Normal, Alpha wave indicator 35 Hz, 100 Hz 0.1 Hz, 1.0 Hz 50 dB rejection @ 50/60 Hz 2 Mohm (Differential) 1000 Mohm (Common mode) 110 dB min (50/60 Hz)

23

Common Mode Input Voltage Range Output Range

Input Voltage Range

10 V (referenced to amplifier ground) 1500 VDC (referenced to mains ground) 10 V (analog) Gain 5000 : Vin 2 (mV) Gain 10000: Vin 1 (mV) Gain 20000: Vin 0.5 (mV) Gain 50000: Vin 0.2 (mV)

3.5 EEG Electrodes

Disc type Silver Silver Chloride Electrodes are used for taking EEG data. These electrodes are disposable and come with their own conducting paste.The corresponding equations for such an electrode can be presented as follows: Ag+ + 1e Ag0(s) Ag+ + Cl AgCl(s) or an overall reaction can be written: Ag0(s) + Cl AgCl(s) + e-

24

Background Theory ackground

4. EEG Signal Acquisition and Event Detection
Before selecting the algorithm, it is imperative to know the characteristics of EEG data for the visually evoked potentials. Ocular data or any other EEG data depends largely on the subject and placement of electrodes. This section deals with analysis of ocular data.

4.1 Placement of Electrodes 1

To get desired ocular data, placement of electrodes is very important. Data has been acquired using disposable silver-silver chloride electrodes and plug in type of connections. Disposable electrodes provide with the flexibility of adjusting electrodes the according to the need of the experiment. Figure below depicts the placement of electrodes on a subject.

Fig. 15 Placement of Electrodes

Connections of electrodes with EEG Machine Ground1, Ground2 Ground2--- Ground FP1, FP2--- Vin+ Reference1, Reference2--- VinReference2

4.2 Eyeblink
During blinking of an eye, eye muscles contract and expand. This results in an eye asymmetric data with the values moving up at some slope and then again settling to base level with some different slopes. During eyeblinks, time index for attaining peaks in both FP1 and FP2 is almost the same. e

25

Fig. 16 Eyeblink Data

4.3 Eyeball Movement

During eyeball movement, same contraction and expansion of muscles takes place but there is a difference between the times of attainment of peaks. One of the peaks occurs before the other according to the movement. Both types of data can be analyzed by finding the time index when slope becomes highest and then using this index with certain threshold to check whether a blink has occurred or eyeballs have moved in some direction. One of the major consideration in the experiment was that of filtering of data, as data was corrupted with significant amount of noise because of the use of make shift electrodes. Presence of power line frequency and noise can result in deviation of results from the desired targets. Special care should also be taken so as to use a type of filter which doesnt provide a large amount of delay element and is also quite computationally light since real time analysis of data needs to take place. During real time analysis and operation, aim is to provide minimum computational delay during analysis so that data loss is very low and data correspondence with the event is quite high.

26

Fig. 17 Movement towards Left

Fig. 18 Movement towards Right

27

Fig. 19 Sequence of Right and Left Movements

Pan and Tompkins[7][8] in their research paper on A Real Time QRS Peak Detection Algorithm have presented a simple and computationally less intensive filter which can be used for filtering the data. It is composed of a mixture of two cascaded low pass and high pass filters and effectively reduces the power line frequency components as well as the noise components. 4.4.1 Low Pass Filter The transfer function of second order low pass filter is

4.4 Filtering

The amplitude response is given by

1 1 |
2

|
2

/2
2 6 12

,where T is the time period. The difference equation for this LPF is given by

,where cutoff frequency is about 11 Hz and gain is 36 with a delay of 6 samples.

28

4.4.2 High Pass Filter The design of high pass filter is based on subtracting the output of a first order low pass filter from an all pass filter. The transfer function of such a high pass filter is given by

1
The amplitude response is given by

32 1 cos 16

256

,where T is the time period. The difference equation for this LPF is given by 32 16 32

,where low cutoff frequency is about 3 Hz and gain is 32 with a delay of 16 samples.

Fig. 20 Result of Filtering

29

4.5 Algorithm
Flowchart below shows the overall algorithm for this project:

a) Fig. 21 Flow Charts a) Main Process (MATLAB)

b) b) Eyeblink Detection and Eyeball Tracking

30

4.5.1 Main Process Data acquisition is started in C++ as soon as the start push button is pressed in the GUI. MATLAB waits for the first epoch of data and when it receives its first epoch it starts the analysis process. After the analysis process is over cursor is initialized and according to the results of the analysis, cursor performs blink and movement operations. 4.5.2 Data Analysis Figure 21 b) shows the flowchart for data analysis. It is a basic skeleton which consists of all the major operations that need to be performed for detecting eyeblinks and eyeball movement. Following are the major tasks that need to be performed: Function Fitting: Function fitting is performed using sum of two Gaussians model: f(x) = a1*exp(-((x-b1)/c1)^2) + a2*exp(-((x-b2)/c2)^2)) Before choosing the model, we should keep in mind that the model should be such that information is not lost and the characteristic of data is also maintained. Figure 22 shows the function fitting on eyeblink data from FP1. Calculating first derivative (point-wise): A point-wise first derivative needs to be calculated to be able to locate the local maxima and minima which prevents the usage of offset and onset calculations. This method is quite fast as after performing function fitting derivative can be easily calculated. Point-wise derivative is also called running derivative of a time-series and is one of the very important analysis tools in ECG and EEG data analysis. Locating the Peaks: Peaks on the derivative data can be located by a simple analysis method. First all the differential values above a calculated threshold are arranged in a matrix. Then all the negative values are arranged and all the positive values are sorted according to their incidence on time-axis. Now an experimentally calculated window of 200ms is used to group all the maxima and minima values which lie in this window. Maximum and minimum values are then taken out of this group, which give the peaks in the ocular data. Here minima values are just minimum in terms of their numerical value (least negative value) but in physical sense these minimum values are maximum speeds of any change in muscular tension while moving the eyeball or blinking.

31

Fig. 22 Function Fitting Using Curve Fitting Toolbox

While calculating the peaks in a data, a window of half the length of epoch is used. There can be cases when the data gets divided in between the two windows. In such cases, instead of alternate negative and positive values of local maxima, there can be only one positive maxima in one window and a negative maxima in other window. To overcome this problem, a shifting window is used when such a condition arises. In this way, loss of characteristics in the data is minimized. Time index evaluation for detection: Data presented in section IV (background theory) has some characteristics which are quite common in all the three cases. There are increasing voltage levels with some slope and decreasing voltage levels with different slope. Computation time can be greatly reduced if single detection algorithm is used for all the three cases. Since the increasing and decreasing characteristics of the data are similar, peaks can be detected easily in all the three types of data. The indices of peaks can be used to differentiate between the type of muscular movement. It can be seen that peaks occur at almost the same time in blinking while there is a delay in one of the peaks when eyeball movement occurs. This information can be used to complete the detection.

32

Cursor Initialization and Movement: Cursor initialization is done using a simple code in MATLAB which calls the java class object for initializing mouse pointer. After initialization, from the data that we have got and analysis we have done, it is possible to guide the cursor in horizontal direction and that too in discrete steps because the data doesnt contain any information on how much eyeballs have travelled in one particular direction. To accomplish this task of continuous tracking more electrodes need to be placed around the eye, particularly below the eye and on one side of the eye. This would make the computations more cumbersome and therefore, in this technology demonstrator only discrete cursor movements have been done.

33

Project Work
Stage 1: Data Acquisition
BIOPAC EEG Data Acquisition System needs to communicate with the computer to transfer data which can be utilized for analysis purposes. One of the major drawbacks with this system is that data acquisition with other interface programs can be done only in serial programming. This means that during data analysis, loss of data can occur because while completing the data analysis part, data acquisition lags. To overcome this problem, data acquisition can be done using Dev C++ and data analysis can be done using MATLAB. A proper communication network should be established between MATLAB and C++ so that there can be no loss of data and MATLAB can also acquire the data as soon as analysis of previously acquired data completes. Following are the main components of data acquisition: MP150acq.exe MATLAB GUI

1. MP150acq
It is an executable file which has been developed in Dev C++ for data acquisition. This is the main interface with Biopac EEG Data acquisition system. It calls the dll, header file and library files necessary for interface with the EEG device. It keeps on waiting for the interrupt from MATLAB GUI. As soon as the star button is pressed with mode selected to realtime, it reads the acquisition file which is again generated by MATLAB as soon as the acquisition parameters are set. As soon as MP150acq gets a 1 in the interrupt file an infinite loop breaks up and data acquisition starts. Data gets stored in two separate text files: fp1data.txt and fp2data.txt. Another text file start.txt contains the number of epochs that have been written on the file and this text file acts as a counter for MATLAB. The counter tells MATLAB to wait if the value of its own index counter exceeds the value in start file. Until the value in start file is zero, MATLAB doesnt start acquiring data. In this way communication gets established between data acquisition and data analysis. Description of files necessary for MP10acq.exe: mpdev.dll: Communication and data acquisition is established with BIOPAC MP Systems by calling this dll file. It is created and distributed by BIOPAC systems. mpdev.h: A header file for MP Systems and contains a list of all the important functions necessary for setting up the dll file. mpdev.lib: Library file for MP systems MP150acq.cpp: C++ source file which can be edited according to our need. Loads the library file and dll files for data acquisition.

34

Graphical User Interface

Data Acquisition

Data Analysis (MATLAB)

Data Acquisition (C++)

BIOPAC MP150 System

Counter

Fig. 23 Block Diagram of Data Acquisition and Communication

2. MATLAB GUI
Data analysis is done in MATLAB environment. The GUI performs various important functions for data acquisition. Acquisition parameters are set in MATLAB using Set Acquisition Parameters drop down menu. When start push button is pressed, the value in interrupt text file goes 1. This value acts as an interrupt for C++ Program to start data acquisition. Then MATLAB GUI goes in a loop to check if first sample of data has been acquired. Since it can become an infinite loop if data acquisition fails or executable file is not working, a check was put inside the loop. If the loop goes on for more than 10 seconds without breaking, then the GUI automatically closes and error dialog box for the same appears. This GUI also prepares .mat files for whole data acquired and also for individual epochs of data. Therefore data acquisition takes place in following steps: Switch on the data acquisition device Run MP150acq.exe Start MATLAB GUI, choose mode=realtime and set acquisition parameters Push start button on MATLAB GUI Data Acquisition Starts.

3. Trial Run
Initial values of communication files: execfile.txt=0 start.txt=0 35

Fig. 24 Output of .exe at start

Start Push Button is pressed execfile.txt=1 acq.txt=5 1000 1(runtime, frequency, length of epoch in seconds)

Fig. 25 Output after end of acquisition

36

After data acquisition is completed start.txt=5 .mat files created: fp1data_1, fp1data_2, fp1data_3, fp1data_4, fp1data_5, fp2data_1, fp2data_2, fp2data_3, fp2data_4, fp2data_5, fp1data, fp2data

37

Project Work
Stage 2: Eyeblink Detection and Eyeball Tracking
For eyeblink detection and eyeball tracking, MATLAB is used as a programming platform. When an epoch of data from the BIOPAC machine is available, analysis starts in MATLAB. For nonlinear regression, Curve Fitting Toolbox is used and a .m file generated from the toolbox is modified to find the differential of the data.

1. List of Main Functions

avgresults locmaxmin maxminanalyse sort1 trackingeye RunDeriv

The complete code is included in Compact Disk (CD) attached with this report. Following is a brief description and demonstration of these functions.

2. Calculation of Differential Data avgresults

2.1 Purpose and Prototype This function generates differential of the data using non-linear regression technique. Sum of two Gaussians is used as the model that needs to be fitted. Following is the prototype of this function: diff = avgresults(data) data is the EOG data diff is the differential of EOG data 2.2 Run and Demonstration >> diff = avgresults(data); >> subplot(2,1,1) plot(fp1data) subplot(2,1,2) plot(diff)

38

Fig. 26 Differential of Data

3. Locate Maxima - locmaxmin

3.1 Purpose and Prototype This function locates initial maxima and minima in the differential of data. Following is the prototype of this function: maxmin = locmaxmin(diff,threshold_upper, threshold_lower) diff is the differential of EOG data threshold_upper is the threshold for positive differential values threshold_lower is the threshold for negative differential values 3.2 Run and Demonstration >> maxmin = locmaxmin(diff,0.002, 0.002) >> size(maxmin) ans = 178 2

39

4. Locate Final Peaks - maxminanalyse

4.1 Purpose and Prototype This function locates final maxima and minima in the differential of data. The values returned from this function are the maximum speeds of eyelid opening and closing. Following is the prototype of this function: vmaxmin = maxminanalyse(maxmin,window) window is the number of samples around which only one maxima can occur. Study of ocular data reveals that this value can be taken to be 200ms. maxmin is the matrix of initial maxima and minima values. 4.2 Run and Demonstration >> vmaxmin = maxminanalyse(maxmin,200) vmaxmin = 0.0048 -0.0048 895.0000 995.0000

>> figure, hold on >> plot(diff) >> plot(vmaxmin(:,2),vmaxmin(:,1),'*r')

Fig. 27 vmaxmin on differential data

40

Fig. 28 Erroneous Data

To correct such errors a shifting window is used which shifts when size of vmaxmin is odd or one. In this way whole data is collected in a window and then analysis can take place in a better way.

5. Sorting time series sort1

5.1 Purpose and Prototype This is a complementary function to sort the vectors as required. This function has two forms. If number of input argument is one, then sorting takes place according to the first column and if input arguments are two, then sorting takes place according to the second column. Following is the prototype of this function: a=sort1(maxmin,d) maxmin is an array of maxima points in derivative data d is an optional parameter and is number of rows of vector maxmin 5.2 Run and Demonstration >> maxmin=[1 3;5 2;2 7]; >> a=sort1(maxmin) a = 1 3 2 7 5 2 >> a=sort1(maxmin,3) 41

a = 5 1 2 2 3 7

6. trackingeye
6.1 Purpose and Prototype This function finds out the blinks, left movement and right movement in the data. It compares the indices of peaks in FP1 and FP2 data and then according to the algorithm, returns the values for all types of movements. These values are checked and cursor is commanded to work accordingly. Following is the prototype of this function:
[early late blinks c1,c2]=trackingeye(vmaxmin1,vmaxmin2, last1, last2]

It outputs 1 if a movement has occurred or not. It checks various cases of appearance of peaks. Suppose that delay in a peak is quite large that it passes out of the present window but appears in the next window, then last1 and last2 are updated accordingly and they are checked against other peak.

7. runderiv
7.1 Purpose and Prototype This function calculates the running derivative of an unevenly spaced time-series with a flat weighing function. Following is the prototype for this function: RunDeriv=runderiv(Data,WindowWidth) This matlab function has been adapted from a MATLAB function by Eran O. Ofek (http://wise-obs.tau.ac.il/~eran/matlab.html)

8. Cursor Initialization and Movement in MATLAB

The ability to control mouse pointer position, motion and clicks is not available in MATLAB. To work around this issue, Java class java.awt.Robot can be used. Below is a simple example that illustrates how one can move the mouse with MATLAB code using Java functionality. import java.awt.Robot; mouse = Robot; mouse.mouseMove(0, 0); screenSize = get(0, 'screensize'); for i = 1: screenSize(4) mouse.mouseMove(i, i); pause(0.00001); end 42

Project Work
Stage 3: Implementation of Graphical User Interface (GUI)
1. Graphical User Interface
GUI provides a convenient way to perform the experiment. This GUI is built in MATLAB 7.6.0. This GUI provides all the capabilities starting from Normalization to Cursor Clicks and Movement. It is shown in Figure 1. This GUI and its all files are available on the CD attached with this report.

2. Description of Different Controls of GUI

Fig. 29 Main GUI Window

This GUI has the following sections: 1. Cursor Bar: The Cursor bar is the section where mouse movement and clicks occur. It consists of a figure and a text bar. The text bar displays the color of the 43

box selected and gives instructions on when to start the experiment. The figure contains five rectangular boxes of different colors. Mouse pointer is initialized at the blue color box and as the mouse moves and clicks on some other box; the color of that box is displayed in the text box. 2. Start/Stop Control: This section consists of the Start and Stop pushbuttons and a table component that displays the calculated parameters for every epoch of data collected. The parameters displayed are mean voltage, rms voltage and variance of the signal. As soon as the Start pushbutton is pressed data acquisition and analysis starts. Pressing of the Start pushbutton checks whether mode of operation is selected. If selected mode is real-time, then another check is put in place which ensures that the acquisition parameters are set. Violation of any of these conditions produces an error dialog box. The Stop button press executes a GUI component which is shown in figure 2. This is a dialog box which asks user if he wants to quit or not. When yes is pressed, the GUI window closes.

Fig. 30 Exit Dialog Box

Fig. 31 Start Stop Control

3. Plot: Plot section plots the incoming data from FP1 and FP2 channels on the two axes components. This helps the user to visually analyze the data and is meant for visual analysis purposes.

44

4. Menu Bar: This section is contains the menu bar of the GUI. The figure shows all the menu items used in the GUI

Fig. 32 Menu Items

Following is the description of the menu items: Mode: Allows the user to select the mode of operation. When the user clicks on this tab a dialog box appears with two radio buttons pertaining to offline and real-time mode.

Fig.33 Mode Dialog Box

Record: Records the data from FP1 and FP2 channels in a specified MAT file. Record as Text: Records the data from FP1 and FP2 channels in a specified text file. It uses dlmwrite function to write data in text file. Exit: Closes the GUI. Acquisition Properties: Sets the acquisition parameters of the data acquisition device. The parameters that can be set are runtime in 45

seconds, sampling frequency and length of epoch in seconds with default values set at 5, 1000 and 1 respectively. It uses an input dialog box to input the values. Eyeblink Parameters: Sets the eyeblink parameters value for analysis of eyeblinks. The parameters that can be set are lower threshold and upper threshold. It also uses an input dialog box to input the values. Eyeblink: This menu item is used to analyze the eyeblinks of the experimenter so that the eyeblink parameters can be set. The popup menu has two axes for plotting the data and differential of data. As soon as start is pressed, an input dialog box appears and user can set the acquisition parameters for the normalization process. Eyeball: This menu item is used to analyze the movement of eyeball of the experimenter so that the thresholds for cursor movements can be set. The popup menu has two axes to display the data from the two channels and three radio buttons out of which one needs to be selected and eyeball moved in the selected direction to start the normalization process. Set parameters push button sets the parameters for acquisition. As start is pressed, data acquisition starts; mean voltage, rms voltage and energy spectrums are stored in the corresponding variables according to the direction selected.

Fig. 34 Eyeblink Normalization

46

Fig. 35 Eyeball Normalization

Help: Opens the help file of the GUI. Documentation: Opens the documentation file of the GUI.
Table 6 Keyboard Shortcuts of Menu Items

Menu Item Mode Record Record as Text Exit Acquisition Properties Eyeblink Parameters Instructions Documentation Eyeblink Eyeball

Keyboard Shortcut Ctrl + M Ctrl + S Ctrl + T Ctrl + X Ctrl + A Ctrl + E Ctrl + H Ctrl + D Ctrl + O Ctrl + N

47

Practical Application
Human Computer Interface
1. Introduction
Human Computer Interface is a technology demonstrator to establish the usage of Visually Evoked Potentials in guiding a cursor. These types of technologies are in advanced stages of development in some countries because they provide a person with severe motor disabilities a way to communicate with machines and computers. These types of machines are often aimed at assisting, augmenting or repairing human cognitive or sensory-motor functions. They have also gained widespread acceptance in neuroprosthetics applications that aim at restoring damaged hearing, sight and movement. U.S. Military is has been exploring applications for such devices, to enhance troop performance and to communicate with the adversaries. A documentation report and all the necessary files and complete codes are available on the CD attached with this report.

Fig. 36 Experimental Setup

2. Main Features
Non-invasive type of technique, so no implantation is necessary. Fast and robust method of controlling cursor movement as compared to image processing based methods. Offline Average Computation Time of 0.0007 seconds and Online Average Computation Time of 0.0157 seconds. Minimal number of electrodes required to control the mouse pointer 48

Lots of future prospects for further improvements.

3. Block Diagram of Human Computer Interface

User Commands

Data Acquisition

Graphical User Interface

Eyeblink Detection and Eyeball Tracking Normalization

Cursor Initialization

Cursor Clicks and Movement

4. Requirements
Following are the software and hardware requirements: 4.1. Hardware Requirements 1. Computer at least 1.3 GHz and 1 GB RAM 2. BIOPAC Data Acquisition System 3. EEG Electrodes 4.2. Software Requirements 1. 2. 3. 4. Operating System: Windows XP MATLAB 7.6 with Curve Fitting Toolbox MP150acq.exe All necessary MATLAB codes for data analysis and acquisition.

5. Operating Instructions
5.1. Starting the System 1. First turn on the computer. 2. Connect the BIOPAC machine to the Ethernet port. 3. Wait for some time for connection. 49

4. Start MATLAB. 5. Start GUI named BTP2. 5.2. Normalization 1. Go to Normalization and Click on Eyeblink 2. Observe differential of eyeblink data and left, right as well. 3. Set eyeblink parameters by pressing Ctrl + E or Settings Set Eyeblink Parameters 4. Go to Normalization again and Click on Eyeball 5. Take the data by clicking on the radio button. Set a particular value of thresholds and then observe the peaks by moving your eyeball in the specified direction selected by radio button. 6. Set the parameters before pressing acquire. 5.3. Running in Real-Time Mode 1. Select the mode by pressing Ctrl + M or File->Mode 2. Set acquisition parameters by pressing Ctrl + A or Settings-> Acquistion Parameters 3. Start MP150acq.exe 4. Push Start button on MATLAB GUI and observe the cursor movement as you move your eyeballs in the desired direction and mouse click as you blink your eyes. 5.4. Running in Offline Mode 1. Select the mode by pressing Ctrl + M or File->Mode 2. Press Start button. Load the .txt file that contains this type of data. 3. Observe the data that has been preset and relate it to cursor clicks and blinks.

6. The Experiment 6.1 Instructions for Cursor Control

1. Cursor movement is a discrete movement. When user wants to move cursor to the left, he/she should see towards extreme left and then come back to centre position. If he/she wants to go left again, then perform the same thing, otherwise move your eyes to the right side. 2. Blink eyes for cursor clicks.

50

Future Work
1. The project can be extended for two dimensional cursor controls. 2. With parallel programming capabilities, more electrodes can be used which can detect the signal with other algorithms that might prove fast. 3. This technology demonstrates the capability of the technique to control mouse cursors. The developed programs can be run in the background and whole computer can be controlled using such technology. 4. Time factor is crucial for real time applications. The code (refer enclosed CD) can be made more efficient to execute large algorithms involved in least possible time. 5. The idea can be used to detect sleep events and level of alertness in a person by measuring the dynamic parameters like mean speed of eye lid closure, stability of the eyeball at the farthest left corner etc. This experiment would prove to be a cornerstone in development of handheld portable devices for measuring alertness levels and prevent road accidents due to sleep events.

51

Conclusion
The project Design and Development of Human Computer Interface Based Controllers was completed successfully. I implemented all the algorithms and provided a very useful practical application in the field of biomedical signal processing. My system works in robust and reliable way and is successful in recognizing movements of the eyeball and eyeblinks. The system has been provided with the capability to normalize and hence, this system is invariant to different persons. A data acquisition interface has been made which is able to communicate between two different compilers and acquire data while analysis is going on as well. Various checks have been placed that help the system to be stable and versatile. Single algorithmic approach has been chosen for identification purposes. This makes the program fast and computationally light. The provided Graphical User Interface (GUI) is very interactive and easy to operate. All the instructions are clear and users can easily perform the desired task in this project.

52

Bibliography and References

1. Yuanqing Li, Chuanchu Wang, Haihong Zhang and Cuntai Guan, An EEG Based System for Two Dimensional Control, International Joint Conference on Neural Networks (IJCNN 2008) 2. Sanjay Gupta and Harvinder Singh. Preprocessing EEG Signals for Direct Human-System Interface, IEEE 1996 3. Aleksander Kostov and Mark Polak "Parallel ManMachine Training in Development of EEG-Based Cursor Control" IEEE Transactions On Rehabilitation Engineering, VOL. 8, NO. 2, June 2000 4. G.A.F Seber and C.J. Wild. Nonlinear Regression. New York: John Wiley and Sons, 1989. 5. R.J.Oosterbaan, Frequency and Regression Analysis. In: H.P.Ritzema (ed.), Drainage Principles and Applications, Publ. 16, pp. 175-224, ILRI, Wageningen, The Netherlands, 1994 6. Curve Fitting Toolbox, http://www.mathworks.com/products/curvefitting/ 7. Pan and Tompkins, A Real Time QRS Peak Detection Algorithm IEEE Transactions on Biomedical Engineering, March, 1985 8. Rangaraj M. Rangayyan, Biomedical Signal Analysis: A Case Study Approach.Singapore: John Wiley and Sons, 2005 9. http://en.wikipedia.org/ 10. Kenneth Levenberg (1944). "A Method for the Solution of Certain Non-Linear Problems in Least Squares". The Quarterly of Applied Mathematics 164168. 11. http://www.biopac.com/

53