Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
AUTOMATED SYSTEM
This thesis is submitted as partial fulfilment of the requirements for the award of the
Bachelor of Electrical Engineering (Hons.) (Electronics)
JUNE 2016
UNIVERSITI MALAYSIA PAHANG
Certified by:
_____________________________ _____________________________
SUPERVISOR’S DECLARATION
I hereby declare that I have checked this thesis and in my opinion, this thesis is
adequate in terms of scope and quality for the award of the degree of the Bachelor
Degree of Electrical Engineering (Hons.) (Electronics).
Signature :
Name of Supervisor : DR. MOHD ZAMRI BIN IBRAHIM
Position : SENIOR LECTURER
Date : 27/05/2016
iii
ii
STUDENT’S DECLARATION
I hereby declare that the work in this thesis is my own except for quotations and
summaries which have been duly acknowledged. The thesis has not been accepted for
any degree and is not concurrently submitted for award of other degree.
Signature :
Name : THUM WEI SEONG
ID Number : EA12056
Date : 27/05/2016
vi
66
6
ACKNOWLEDGMENT
I thank my fellow friends and course mates for helping me to do the data
analysis. Thank you for supporting me spiritually and cheer me up always.
Last but not the least, I would like to thank my family: my parents and to my
brothers for supporting me spiritually throughout writing this thesis and my life for
general.
vi
77
7
ASTRACT
In year 2016, the population number of Malaysia was expanded to 30 572 466
peoples. This is an increasing of 1.51 % (455 964 people) compared with population
of 30 116 502 in year 2015. At the beginning o f 2016, total number of older people
with age above 64 years old is 1,520,369 (5% of total population). While in December
2012, there were total 359,203 disabled people under registration with Department of
Social Welfare. They all have trouble in doing da ily stuff and they hard to switch on
some electronic device too. The aim of this project is to use Brain-Computer Interface
(BCI) systems to interpret users’ intentions into command to control home automated
systems. This BCI system enable brain signals to activate or turn on device by using
NeuroSky MindWave Mobile that utilizes Electroencephalograms (EEGs) used to
decode human intentions. This BCI system is developed by using MATLAB graphical
user interface (GUI). Then, IP Webcam is used for live video streaming and then
undergo image processing using MATLAB to set the path where those command
should direct to. As for the BCI system output, Arduino UNO will be used as output
controller to activate electronic devices. Besides that, data analysis on the user
concentration based with gender and condition is conducted to test the impact of
different conditions on people concentration performance. Condition for this data
analysis are relax, happy and stress condition. Based on the results of data analysis,
people in happy condition are able to concentrate 36.67% faster than people in stress
condition. People in relax condition able to concentrate much faster which is 73.33%
compare to people in stress condition. By having this BC I system, conventional light
and switch can now be easily replaced and activated by just mind concentration of
user. This can act as cognitive tool to offset the effects of aging and alternative
method for disable people to control electronic devise at home.
888
88
ABSTRAK
Di Malaysia, jumlah orang tua dari usia 50 tahun ke atas adalah lebih daripada
17% pada tahun 2011 dan trend ini semakin meningkat lebih cepat daripada jumlah
penduduk. Penduduk Malaysia telah mencapai 26.640.000 dalam tahun 2006,
bagaimanapun orang yang kurang upaya adalah berjumlah 197.519. Mereka
mempunyai masalah dalam melakukan kerja harian dan sukar untuk menghidupkan
peranti elektronik juga. Tujuan projek ini adalah untuk menggunakan Brain-Computer
Interface (BCI) untuk mentafsir isyarat otak pengguna ke dalam arahan untuk
mengawal sistem automatik rumah. S istem BCI ini membolehkan isyarat otak untuk
mengaktifkan atau menghidupkan peranti dengan menggunakan NeuroSky
MindWave Mobile yang menggunakan Electroencephalogram (EEGs) yang
digunakan untuk memperoleh tahap tumpuan manusia. S istem BCI ini dibangunkan
dengan menggunakan MATLAB Graphical User Interface (GUI). Kemudian, IP
Webcam digunakan untuk live video secara langsung dan menjalani pemprosesan
imej menggunakan MATLAB untuk mene ntukan laluan isyarat. Bagi output sistem
BCI, Arduino UNO akan digunakan sebagai pengawal output untuk mengaktifkan
peranti elektronik. Selain itu, analisis data kepada tumpuan pengguna berasaskan
dengan jantina dan keadaan dijalankan untuk me nguji kesan keadaan yang berbeza
kepada prestasi tumpuan. Keadaan untuk analisis data ini adalah relaks, gembira dan
tekanan. Berdasarkan hasil analisis data, orang dalam keadaan gembira dapat
menumpukan perhatian 36.67% lebih cepat daripada orang berada dalam keadaan
tekanan. Orang dalam keadaan relaks dapat menumpukan perhatian lebih cepat iaitu
73.33% berbanding dengan orang berada dalam keadaan tekanan. Dengan adanya
sistem BCI ini, cahaya konvensional dan suis kini boleh dengan mudah digantikan
dan diaktifkan dengan hanya penumpuan fikiran pengguna.
999
99
TABLE OF CONTENTS
Page
SUPERVISOR’S DECLARATION iii
STUDENT’S DECLARATION iv
ACKNOWLEDGEMENT vi
ABSTRACT vii
ABSTRAK viii
TABLE OF CONTENTS ix
LIST OF TABLES xi
LIST OF FIGURES xii
LIST OF ABBREVIATIONS xv
CHAPTER 1 INTRODUCTION
2.1 Introduction 6
2.2 Research of Invasive BCI 7
2.3 Research of Non-Invasive BCI 8
2.4 EEG brain wave reading headsets 8
2.4.1 Emotiv EPOC Headset 10
2.4.2 Melon Headband 11
2.4.3 NeuroSky MindWave Mobile Headset 12
2.4.4 Comparison 13
2.5 Previous works 13
2.6 Summary 20
vi1
0
CHAPTER 3 METHODOLOGY
3.1 Introduction 21
3.2 Architecture 21
3.3 Hardware 22
3.3.1 NeuroSky MindWave Mobile headset 23
3.3.2 IP Webcam 26
3.3.3 Arduino UNO 28
3.3.4 USB Lamp 29
3.4 Software 29
3.4.1 MATLAB 29
3.4.2 Music Player 31
3.5 Methodology 31
3.6 Summary 35
4.1 Introduction 36
4.2 Experimental Setup 36
4.2.1 Relax Condition 36
4.2.2 Happy Condition 38
4.2.3 Stress Condition 40
4.3 Output Result 42
4.4 Summary 49
5.1 Conclusion 50
5.2 Recommendation 51
REFERENCES 52
vi1
1
LIST OF TABLE
LIST OF FIGURE
2.6 Nokia E72 used for HK (a) and Nokia N8 used for SK (b) 13
4.10 Output of Attention value (a) raw data and (b) smooth data 42
LIST OF ABBREVIATION
EEG Electroencephalography
EMG Electromyography
EOG Electrooptigraphy
HK Hard Key
SK Soft Key
CHAPTER 1
INTRODUCTION
BCI system enabled direct communication between human brain and external
device. An early research of BCI began in the 1970s at University of California, Los
Angeles (UC LA)[5]. The papers published after this research also mark the very first
appearance of the expression BCI system in scientific literature. N icolelis, C hapin, and
theirs colleagues did a lot of laboratories have managed to record signals from monkey
cerebral cortices to operate BCI to produce moveme nt in early 2000s. Monkeys been
22
In year 2016, the population number of Malaysia was expanded to 30 572 466
peoples. This is an increasing of 1.51 % (455 964 people) compared with population of
30 116 502 in year 2015. At the beginning of 2016, total number of older people with
age above 64 years old is 1,520,369 (5% of total population) [10]. While in December
2012, there were total 359,203 disabled people under registration with Department of
Social Welfare. S ince this registration is under volunteers based, so the actual number
of it ought to be much higher than that. They all experiencing difficulty in doing daily
stuff and they even unable switch on the light without anyone else since they are
generally reliant on their relatives.
There are several objectives need to be achieved of this project which is:
For processing part, after the stabilizing of brain signal output, it will
synchronize with the object detect by IP camera. IP Webcam is used for video live
streaming and transmits it into MATLAB. Video will be display in MATLAB and
doing image processing on the video frame by frame. The activation of hardware output
device will be controlled by Arduino UNO board. It will be reacted according to the
object detected and activation command will transmit from MATLAB.
The next chapter deals with the literature review on the research on types of BCI
system and previous works that applied BCI system. C hapter 3 shows the hardware and
software involved in developing the BCI system. This chapter also list out the technique
on the brain signal and image processing. Chapter 4 presents the result of this project
and data analysis on human concentration according gender and condition based. Last
chapter discussed the conclusion and future improvement of this project.
CHAPTER 2
LITERATURE REVIEW
2.1 INTRODUCTION
This chapter will discuss the previous work done by other research. All the
technique to acquire EEG signal from human brain and effect on brain concentration
based on different condition will be further discuss in this chapter. This project is done
based on the problem encounter on the previous work and to improve the current BCI
system. A BCI system is a new communication channel between the human brain and a
digital computer. The goal of BCI system technology was to provide control capacities
to people with severe motor disabilities. An electroencephalogram (EEG) based brain-
computer interface was connected with a Virtual Reality system in order to control a
smart home application[11]. A typical BCI system was composed by the following
blocks which are data acquisition system, signal processing system and commands sent
to an application. For the acquisition system, the most frequently used for recording
method is EEG. Brain activity is recorded from the surface of scalp, so this method is
non- invasive with advantage of portability of the recording system[12]. The signal
processing block processes all the recorded data and transforms the signals in
commands for the application. The system must be fast enough to process the data
record to achieved real-time operation. It also needed to ensure that the extracted
features are not contaminated by electromyography (EMG), electrooculography (EOG)
or some other non-central nervous system artefacts.
77
For invasive BCI, an array of electrodes is planted in the grey matter of the
patient’s brain during a surgery. Those electrodes are directly connected to neurons.
By this way, it offer a high quality for recorded signals of BCI devices, but the
problem is that the body may react to the foreign objects in the brain, creating the
possibility of scar-tissue build- up over the implanted device. Recognition of different
patterns of signals recorded from motor cortex neurons is the key in controlling
robotic arm or neuroprostheses [13]. Direct brain implants were used to treat
blindness. Scientist William Dobelle designed a working brain interface prototype in
1978s. The prototype was implanted into the visual cortex of man blinded in
adulthood. The single-array BCI contained 68 electrodes and the prototype succeeded
in producing phonophenes, the sensation of seeing light without light actually entering
the eye.
Another area of BCI is focusing on motor neuroprothetics. The aim of these BCI
is to either restore movement of individuals with paralysis or provide some special
devices to assist them. Special devices can be different types of interfaces with
computer or even robot arms for control. Previous studies on monkeys (Macaca
mulatta) proved that invasive methods are suitable for creating a prosthetic robot-arm
for control [14]. Natural arm movements can be recorded in populations of neurons
from motor cortex [15]. Moreover, these studies proved that continued training over of
the executed task. In addition, these studies revealed the possibility to develop real-time
controlling systems such as robot arm controlling system. To conclude it, invasive BCI
are currently much more suitable for real- time applications. Recent tests on monkeys
proved that, robotic arms and prostheses can be controlled n real-time with invasive
BCI. Also invasive on humans are a success, one patient can control a cursor on a
monitor, thus the subject can operate a computer only by his thoughts. But, the biggest
problem is the high costing of the required special electrodes array. A special expensive
surgery need to be done to implant those electrodes in the patient’s brain. The durabilit y
of electrodes may decrease, although the patient’s brain is not affected. Scar-tissues
might grow near to the electrodes, then the quality of recorded signal can be altered or it
can be even stop record signals.
88
Figure 2.1: Electrode cap with electrodes placed on users head [1]
99
Non-invasive systems are much cheaper than invasive ones, but the main problems
raised are related to the computational response time fro m the system. While real-time
applications are required for locked- in patients, the non-invasive methods cannot yet
meet such a requirement. Also, these systems offer a very good communication channel
for locked-in patients and can be used on a larger scale because of the reduced price.
Although the patient cannot generate a message very fast, the BCI still allows
communication with other people, which is very important for patients who cannot even
move their eyes for answering questions.
10
1
0
This section will explain in detail and discuss on the training method used for
EEG- based brain-computer interface. Based on previous work and latest research, there
are several types of devices that able to measure brain wave. At here, there are some
devices being picked to make comparison to determine the most suitable for my project.
Emotiv EPOC EEG headset is a sleek, 14-channel, wireless headset that records
brains wave activities and translate them into data for analysis. This device can capture
emotions and thoughts in mind and transfer into readable codes. Emotiv Insight earlier
version (EPOC) was released in 21 st December 2009. It is very comfortable due to its
lightweight. The software that comes with the Emotiv headset provides the following
detection functionality such as various facial expressions, levels of engagement,
frustration, meditation, and excitement[18].This device gives full control of the external
devices that is connected with it. Usage of this device is to help people with disabilities
in their daily life. This device easy to set-up, so handicapped people will find it more
comfortable. On the device are EEG sensors that detect brain neurological activity,
which can be transferred to smartphone via Bluetooth. An Emotiv needed to plug in
11
1
1
USB cable connected to laptop to run a calibration process. User need to relax for 10
seconds for the system to get the baseline reading. F urther, it is wireless technology
which allow user to carry around. But it requires applications that to be able to apply on
other device. The price of Emotiv Insight is quite expensive for public which cost $399
to $499.
The Melon Headband is a wireless brain monitor that uses EEG to measure brain
activity. From the activity, the algorithm can able to detect user‘s level of focus,
relaxation, or meditation. It help user to understand and improve focus across activities
ranging from study to exercise. Starting, Berman and her co- founder Arye Barnehama
met as undergraduates studying cognitive and computer science at Pomona College and
doing research on EEG technology. They are using this on the treatment of ADHD and
dementia in a clinical setting. Melon Headband uses alpha and beta to calculate a neural
focus score and save those data in smartphone apps. These data can be tracked and
correlated to different tags to see the pattern of focus over time. It designed for low
power consumption and with Bluetooth 4.0 with the total cost of $149.00.
12
1
2
2.4.4 Comparison
There is some research on changes of EEG signal based on some condition. The
terms of attention and meditation level are authored by Neurosky EEG information that
comprises of interpretation through eSense meter (Attention and Meditation).There is a
research that quantify the impacts of input gadgets have on player experience through
attention and meditation levels utilizing biosensors estimations, for example, EEG and
its connection to portable hard key (HK) and soft key (SK) input.
14
1
4
(a)
(b)
Figure 2.6: Nokia E72 used for HK (a) and Nokia N8 used for SK (b) [24]
There was 10 participants were all are gamers but they ne ver played the game
prepared before. Among of those 10 participants, there are 5 female and 5 male
undergraduate students at the age group of 19-22 years old. Before the testing began,
game play and its mechanics were explained. Members were then be given 1-2 minutes
to acclimate themselves with the given cell phones. When members were prepared, Video
of their game play, levels of attention and meditation is recorded. Every gaming sessions
conducted around 8-10 minutes while the overall testing session took around
15-20 minutes. Table 2.2 displays an overall effect of general players' attention and
meditation level for arcade game on HK and SK. For the game play on first level, the
15
1
5
finishing time for HK was quicker in contrast with SK. So also, more error rate was
stand up to on SK input.
The conclusion on this research is that for HK's attention level, player appears to
have lower attention rate contrasted with SK, while HK meditation rate is higher in
correlation with SK. This outcome proposes that players required more focus on SK, but
they are ready to play more on HK with less stress. This could be to a great extent
because of HK have kinesthetic criticism as players could touch and feel the catch
physically in compare with SK. For SK, the players will focus more on the keypad and
on the screen as they rely on upon their visual methodology more for SK in examination
with HK [24]. As overall, the attention and meditation value will be differ according to
different type of input device, but this research unable to prove that attention and
meditation value do not effected by users’ emotion.
There was another research that about to propose a helpful BCI system, which
sets a proper attention value and eyes blinking to use and activate simple cell phone. By
accomplishing focus state over predefined level, the user is able to perform essential
operations like telephone call, sending SMS, email with predefined content or play
voice message [25].
16
1
6
Figure 2.7: Value of brainwaves acquired from MindWave Mobile device [25]
EEG Analyzer acquired information of attention and meditation level from all
time. Recorded attention level was mostly around 40% to 50% under normal condition
user monitoring a cell phone. When somebody concentrates on a particular action or
tries to be concentrated, the attention level is can be achieve to a high value of
concentration that higher than 70%. But then normally drained individ ual will have
issues with accomplishing attention level higher than 50% or 60%. Next, strength of eye
blinking quality differs somewhere around 38% and 60%, yet now and again can be
higher and its worth can be more than 80%.
The predefined value for this research is 75% of attention value, since it can be
achieved by a normal individual and it is not general focus level. Higher or lower center
level may bring about improper action with the client. Besides it, eye blinking is
counted as a controller input too. It used double blinking instead on single eyes blinking
due to double blinking is not a normal human behavior. S ingle blinking is just like
human regular blinking, so it may cause unwanted error happened. As overall, this
research is talking about how to fully control over one device, but it cannot be control
over more than one output device.
remind the user when she/he is losing focus and eventually help the user to rapidly
recover his/her concentration on the current content.
There are 20 participants in this research, which are 10 male and 10 female in
the age group of 17-23 years old. This research used EEG sensor of NeuroSky
MindWave Mobile headset to acquire user attention value and a webcam to capture user
face to determine whether user is focusing the text on screen or not.
18
1
8
Figure 2.9: The user is reading on the computer’s screen with Mindwave Mobile
headset [26]
This ReadGoGo! utilized two input sources which are from the webcam and
from EEG sensor, to figure out whether the user is concentrating on the text on the
computer’s screen [26]. The framework helps the user to rapidly re-concentrate on the
object after psyche meandering or physical distractions. As overall, this framework can
eventually help the user know about their concentration levels which thus help them to
re-concentrate again, but it do not provide the exact way for user how to re-concentrate.
There is a field study on BCI system conducted at Graz, Austria which total 99
healthy people in the age group of 16-60 years old participated in this study. This study
is to test the efficiency of its experimental testing. The EEG was recorded with gold
anodes from two bipolar channels over the right- hand and foot representation regions
(C3 and C z) [27]. The head is viewed from above and nose pointes to the top of the
page.
19
1
9
Figure 2.10: Electrode positions (C3 and Cz) for EEG measurements. [27]
Figure 2.11: Depicts the hardware and software architecture of the portable BCI system
[27]
Participants will able to achieve the target if the brainwave acquired from EEG
sensor is match with the desired pattern. A large group outside the research center been
participated to test the efficiency to operate the BCI system with just 20 -30 mins of
20
2
0
training by using only two bipolar EEG inductions. As overall, t here is around 80%
accuracy of the BCI system after 20-30mins of training, but this is only on two parts
which is right hand and foot. A specific point of human brain is required to acquire
exact EEG signal, it will cause error if getting the wrong position.
2.6 SUMMARY
Based on previous works, there are some testing on the effect on EEG signal
based on difference factors, but there do not have a testing on mental states on human
being that affect the concentration. There were BCI system developed to remind user to
concentrate, but do not provide a proper guide for user how to re-concentrate. The more
channels of electrode needed for brainwave detection may cause more noise. Wrong
position of electrode will caused failure in BCI system. By having the comparison of
difference EEG headset, NeuroSky MindWave Mobile headset is more suitable for my
project as there is only one channel of electrode needed and low costing.
CHAPTER 3
METHODOLOGY
3.1 INTRODUCTION
This chapter will discuss the detail steps from beginning with data acquisition,
pre-processing, processing, and final with output parts of this BCI system. All the
software and hardware parts involved are NeuroSky MindWave Moble headset, IP
Webcam, Arduino UNO board, US B Lamp, and MATLAB. BCI system will be
developed by utilized all the software and hardware parts.
3.2 ARCHITECTURE
3.3 HARDWARE
Double Blinking Red Low Battery Need to replace the MindWave bettery
This device must be set up correctly to ensure brain wave can be fully detected.
Orient the MindWave headset with the forehead sensor arm on left hand side. Make
sure the front sensor make contact with the forehead, but if the sensor does not make
contact with the forehead, remove it and readj ust the adjustable head band and sensor
arm. The Sensor Tip must maintain steady skin contact in order to properly measure the
brainwave. Then, the flexible Ear Arm should rest behind the left ear and then clip the
Ear C lip onto the earlobe. Two metal contacts on the inside of both sides of the ear clip
make sure direct clip on the earlobe. Any hair or obstructions should be removed to
avoid any error happened[28]. The complete setting up of NeuroSky Mindwave Mobile
headset on human head is shown as Figure 3.4 below.
25
The device able to measures the raw signal, power spectrum (alpha, beta, delta,
theta), attention level, mediation level and blink detection. All different brainwa ve types
represent different human mental states and condition. Table 3.2 above shows the
frequency range of brainwave type and its representative mental states.
The outputs of MindWave Mobile is raw EEG data as well as the eSense™ (i.e.
Attention, Meditation) in scale of 1 to 100 [23]. In order to have interaction connection
with computer, The NeuroSky MindWave Mobile headset must paired with the
26
computer by using Bluetooth. Serial COM port name of headset was identified using
Bluetooth software in order to connect or interface with some applications. Default port
setting for Bluetooth connectio n of NeuroSky MindWave Mobile with computer should
be same as Figure 3.5 below.
3.3.2 IP Webcam
In the IP Webcam mobile apps, user can customer some setting of it regrading of
its purpose. Users are able to set the resolution and quality of the video for higher
definition. The highest resolution of video can reach to 1280x720p and the highest
transmission of video can reach to 16000kbit/s. Besides that, it has password security,
user will required to log in with correct password in order to access the webcam.
Figure 3.7 below show the video live streaming monitor in mobile device.
Once the video is streaming, the IP address is shown at the bottom of the
monitor. By typing the same IP address into any internet browser or Android device to
view the video live streaming.
28
USB lamp is powered by any 5V USB port such as laptop, power bank or any
other US B charger. The slim design and bendable body is convenience for any usage.
The Average life of it is around 10,000 hours. US B Lamp in this project is act as the
output. Its interaction with other components will be discussed further in this chapter.
3.4 SOFTWARE
The software involved in this project is MATLAB and Music P layer. MATLAB
will act as signal and image processing, and then Music P layer is act as output device.
All those software will interact with hardware to develop the BCI system. The whole
BCI system will run the program flow according to the flow chart in Figure 3.10.
30
Start
Detect image
from IP Webcam
No
No
Green Ball ? Red Ball ?
Yes Yes
Turn ON Turn ON
Indicator Light Indicator Light
(USB Lamp) (Music Player)
No No
No No
Attention Value > 80 Attention Value < 5 Attention Value > 80 Attention Value < 5
3.4.1 MATLAB
interaction connection with Arduino UNO. Besides that, Image processing toolbox also
required to conduct some image processing during processing part. Then, MATLAB
GUI will be used to develop the whole BCI system to make it more users friendly.
The Music P layer used in this project is Window Media P layer (WMP). WMP is
a media player and media library application created by Microsoft that is utilized for
playing sound, video and review pictures on computer. This music player support many
formats of audio, video, and even picture files, such as .wma, .wmv, .avi, .mpeg, .mpg,
.mp3 and .vob.
3.5 METHODOLOGY
NeuroSky MindWave Mobile headset paired with computer in order for data
transmission. The connection of headset and computer will be through Bluetooth
connection. NeuroSky MindWave Mobile headset needs turn into paring mode in order
to have connection with computer. During the pairing process, under the device column,
need to check that box to enable the data transmission and identified the COM Port
name of headset too.
Figure 3.11: Bluetooth Setting for data transmission of MindWave Mobile headset
32
After the pairing, MATLAB can now start acquire brainwave signal from
MindWave Mobile headset by calling the ThinkGear library of NeuroSky MindWave
which established based on MATLAB and Arduino IDE with C++. The outputs of
MindWave Mobile is raw EEG data as well as the eSense™ (i.e. Attention, Meditation)
in scale of 1 to 100 [23]. This project is mainly focus on the Attention value from the
user. S ince the raw output data is a bit unstable and fuzzy and hard to prove whether
user is concentrating. Then, 5-points moving average smoothing method used by C urve
Fitting Toolbox in MATLAB is applied to stabilize the data for better analysis [33].
Smooth function in MATLAB is used to smooth the Attentio n value acquired from user
brain according to the Formula 3.1 below.
�
��(��) = (�(𝒊 + ��) + �(𝒊 + 𝑵 − �)+ ⋯+ �(𝒊 − ��)) 3.1
���+�
Where ��(i) is the smoothed value for the ith data point, N is the number of
neighboring data�points on side of ���(i), and 2N+1 is the span. In this project, the value
of span is set to 5 for better smoothing effect. 5-points moving average technique can be
prove user is really concentrating and will not effected by sudden spiking of the
concentration value.
For the image process ing part, IP Webcam will start capturing image and live
video streaming on the MATLAB GUI. Every frame of the image capture by IP
Webcam will be undergoing image processing. Every image having the original color
which consists of red, green and blue (RGB). The RGB value of image will then turn
into gray scale image according the formula 3.2 below.
�.���
�× 𝑹 + �.����× 𝑮 + �.����× � 3.2
After convert the RGB value of image into grayscale, “imfindcircles” function
that available in MATLAB Image Processing Toolbox MATLAB will be used to detect
any circular object. The “imfindcircles” function using C ircular Hough Transform
(CHT) based algorithm to determine any circles in image. This algorithm having object
polarity detection that identifies the circular objects which brighter or darker then
33
background. The purpose that turn image into gray scale is to make the circular object
able be detected easily. The CHT technique will be detect circular object based o n the
range of the radius given. By having the radius, center of circular object can be
identified easily.
�
���� Center of the circle of radius 𝑟�
��𝑥
�
������
��� Center of the circle of radius �����������
There is two output of this project which is US B lamp and music player. For the
output hardware USB lamp, there is some circuit connection. The components used for
that circuit connection are 9V battery, PNP type transistor, two 220Ω resistors, two
LED indicators light and a US B holder. In this project, the Arduino UNO board will be
the output controller that interfaced with MATLAB. As mention before, Arduino UNO
board can be programmed by MATLAB coding by installing the Arduino Support
Package of MATLAB. By having that support package, all the output and input pin of
Arduino UNO board is now can be controlled by MATLAB.
If there is any green circular object detected, the pin 4 of Arduino UNO will be
activated to trigger the indicator light of US B lamp. But if red circular object detected,
pin 2 of Arduino UNO will be activated to trigger the indicator light of music player.
Then, while green circular object is detected and at the same time attention value of user
is higher than 80, then pin 7 of Arduino UNO will be activated to turn on the US B lamp.
In the other hand, if red circular object detected and attention value higher than 8 0, then
music player of computer will be turn on by using “winopen” function in MATLAB.
This “winopen” able to open any files which perform the same action by double click
any files in Windows Explorer program. F igure 3.13 below shows the circuit connectio n
of USB lamp with Arduino UNO board.
3.6 SUMMARY
This chapter discussed about the architecture of the while BCI system which
combined with hardware and software parts. All the software and hardware parts
involved are NeuroSky MindWave Moble headset, IP Webcam, Arduino UNO board,
USB Lamp, and MATLAB 2015b. Attention raw data values were stabilized by using
5-points moving average technique by Curve F itting Toolbox in MATLAB. For the part
of image processing of image, c ircular object can be easily detected by using CHT
technique. Threshold value of RGB will be setting to only detect green and red circular
object. O utput of the BCI system will be controlled by Arduino UNO b oard which
having direct interface with MATLAB.
CHAPTER 4
4.1 INTRODUCTION
This section discuss about the data analysis about the human concentration
based on gender and condition. There will both male and female participants take part in
this data analysis. The conditions take in consideration in this analysis will be relax,
happy and stress condition. The further explanation on experimental setup and data
analysis will be discussed in this chapter.
For this project, there are total 8 people in the age group of 21-23 years old
participated in this data analysis, which are 4 male and 4 female. Participants will
required to undergo every condition such as relax, happy and stress before doing the
data analysis to test the impact on the human concentration.
The first one is relax condition. Participate required to do nothing and resting for
5 minutes before taking the data analysis. While participant is resting, their brainwave
signal is continuous display on the MATLAB GUI. Facial expression of participants in
relax condition is shown at the Figure 4.1.
37
100
90
80
Attention Reading
70
60
50
40
30
20
10
0
0 5 10 15 20 25 30 35 40 45 50 55
Time (seconds)
100
90
80
Attention Reading
70
60
50
40
30
20
10
0
0 2 4 6 8 10 12 14 16
Time (seconds)
Figure 4.3: Time taken for attention value to reach 80 after relax condition
100
90
80
Attention Reading
70
60
50
40
30
20
10
0
0 5 10 15 20 25 30 35 40 45 50 55
Time (seconds)
100
90
80
Attention Reading 70
60
50
40
30
20
10
0
0 5 10 15 20 25 30 35
Time (seconds0
Figure 4.6: Time taken for attention value to reach 80 after happy condition
100
90
80
Attention Reading 70
60
50
40
30
20
10
0
0 5 10 15 20 25 30 35 40 45 50 55
Time (seconds)
After solved all the mathematical questions online, all the participants required
to focus immediately to test their concentration. The F igure 4.9 below showing that
attention value of participants unable to reach to 80 within 2 minutes.
100
90
80
Attention Reading
70
60
50
40
30
20
10
0
0 5 10 15 20 25 30 35
Time (seconds0
This project will mainly focus on one mental state which is attention from user.
The raw data of brain signal from user brain is fuzzy and unstable. By applying the 5-
points moving average smoothing technique, the output graph will be very smooth and
stabilized for further brain signal processing. The output result of this smoothing
process is shown at the Figure 4.10 below.
100
90
80
Attention Reading
70
60
50
40
30
20
10
0
0 5 10 15 20 25 30 35 40 45 50 55
Time (seconds)
(a)
100
80
Attention Reading
60
40
20
0
0 5 10 15 20 25 30 35 40 45 50 55
Time (seconds)
(b)
Figure 4.10: Output of Attention value (a) raw data and (b) smooth data
43
For this BCI system, MATLAB GUI is developed act as a signal processor to
generate command based on the inputs received. Starting of the MATLAB GUI will
require user to enter password to access the program. If user forgot the password, click
on the “Forgot Password” to reset a new password.
After entering the main page of BCI system, there will be a “S TART” and
“S TOP” button to activate the program and terminate the program respectively. There is
two column on the left that shown the real-time graph of attention reading from
MindWave Mobile headset and live video streaming from IP Webcam.
By clicking the “START” button, the system will start acquire attention value
from NeuroSky MindWave Mobile headset and IP Webcam is now started to capture
image and video live streaming in the MATLAB GUI by accessing the IP address with
internet connectivity [34]. There is a column that allowed user to enter the trigger value
for activation of electronic device. Default trigger value for this BCI system was set to
80. The real-time graph of attention value and video live streaming from IP Webcam is
shown as the Figure 4.13 below.
Figure 4.13: Real-time graph of Attention value and video live streaming
(a)
(b)
46
(c)
Figure 4.14: Difference between RGB (a), Grayscale image (b) and Circular object
detected from grayscale image (c)
Then if any circular objects are detected, it will identify the original color of that
circular object. In this project will only focus on green and red color. The threshold
value for green and red color are set to ensure the fina l result will be only green or red
circular object. Threshold value for green and red color to be detected is predefined.
At first, red color object will be filter out and turn it into binary image. A binary
image contain only 2 color pixel which is black and white. The F igure 4.15 shows the
steps and result on how to extracts the red circular objects on the image using color
thresholding.
47
(a)
(b)
(c)
Figure 4.15: Original image (a), red object extracted on original image (b) and red
circle object is detected (c)
48
Then whenever green circular object is detected, the indicator light of USB
Lamp and the MATLAB GUI will display “Green Ball Detected”. At this moment, if
the user attention value acquired from MindWave Mobile headset is higher enough and
more than 80, then USB Lamp will be turn on.
Then in case of red circular object is detected, indicator light of Music P layer
will be turn on and MATLAB GUI will display “Red Ball Detected”. Then user
attention value is higher than 80 then Music P layer will be turn on automatically. F igure
4.16 and F igure 4.17 shown the output result of green and red circular object is detected
with attention value higher than 80.
Figure 4.16: Output result of green circular object detected with attention value higher
than 80
Figure 4.17: Output result of red circular object detected with attention value higher
than 80
49
While user is sleeping or meditation, attention value will drop to a very low
value. Once attention value acquired is lower than 5, then all devices such as USB
Lamp and Music Player will be turns off immediately.
4.4 SUMMARY
In conclusion, based on the data analysis, time taken for people to concentrate
will be very long if under stress condition. Then, Time taken will become shorter in
happy condition compare to stress condition. Then, people are able to concentrate easily
if allowed to relax before focus on doing something. For the output result, once green of
red circular object is identified, indicators light of it will be turn on. Then, user mind
concentration if higher than 80 at that moment, US B lamp or music player will be turn
on. Then, based on the results of data analysis, people in happy condition are able to
concentrate 36.67% faster than people in stress condition. People in relax conditio n able
to concentrate much faster which is 73.33% compare to people in stress condition.
CHAPTER 5
5.1 CONCLUSION
BCI system was successfully developed to control home automated system. The
raw EEG signal acquired from human brain are able be interpreted and stabilized using
5-points moving average smoothing method used by Curve F itting Toolbox in
MATLAB. The fuzzy and sudden spiking of concentration value will not affect the real
concentration value. The stabilized attention value is now very smooth and suitable for
data analysis and output device control.
By having the image processing on the image acquired, human can now control
over multiply electrical devices with a symbolic representative object. In this project,
this concept had proved that user able to choose turn on US B lamp and music player.
Although there is only one output signal acquired from NeuroSky MindWave Mobile,
but this signal can be directed depending on what kind of object detected.
For the d ata analysis parts, a proper experimental testing been conducted to
analysis the brainwave signal concentration based on gender and condition. Based on
the data analysis, time taken for people to concentrate will be very long if under stress
condition. Then, Time taken will become shorter in happy condition compare to stress
condition. Then, people are able to concentrate easily if allowed to relax before focus on
doing something.
51
5.2 RECOMMENDATION
REFERENCE
[1] M. Teplan, “Fundamentals of EEG measurement,” Meas. Sci. Rev., vol. 2, pp. 1–
11, 2002.
[3] W. Sa, “Processing and spectral analysis of the raw EEG signal from the
MindWave,” West Pomeranian Univ. Technol., no. 2, pp. 169–173, 2014.
[6] I. Wickelgren, “Tapping the Mind (News Focus),” Science (80-. )., vol. 299, no.
January, pp. 496–499, 2003.
[8] E. Farwell, L.A., Donchin, “Farwell Donchin 88 - first P300 BCI.pdf.” 1988.
[11] M. Govindu and A. Rajaiah, “Automatic Home Control System Using Brain
Wave Signal Detection,” no. October, pp. 889–893, 2014.
[12] C. Postelnicu, D. Talaba, and M. Toma, “Brain Computer Interfaces for Medical
Applications,” Bull. Transilv. …, vol. 3, no. 52, pp. 99–106, 2010.
[17] H. Ekanayake, “P300 and Emotiv EPOC: Does Emotiv EPOC capture real
EEG?,” Web Publ. http//neurofeedback. visaduma. info/ …, p. 16, 2010.
[20] J. W. Matiko, S. Beeby, and J. Tudor, “Real time eye blink noise removal from
EEG signals using morphological component analysis.,” Conf. Proc. IEEE Eng.
Med. Biol. Soc., vol. 2013, no. 2, pp. 13–6, 2013.
[24] K. Chu and C. Y. Wong, “Player’s attention and meditation level of input devices
54
on mobile gaming,” Proc. - 2014 3rd Int. Conf. User Sci. Eng. Exp. Eng. Engag.
i-USEr 2014, no. September 2014, pp. 13–17, 2015.
[25] J. Of and M. Informatics, “Brain-Computer Interface for Mobile,” vol. 24, 2015.
[27] G. Pfurtscheller, “How Many People are Able to Operate an EEG-Based Brain-
Computer Interface (BCI)?,” vol. 11, no. 2, pp. 4–6.
[34] S. Saha and S. Neogy, “A case study on smart surveillance application system
using WSN and IP webcam,” Appl. Innov. Mob. Comput. (AIMoC), 2014, no.
February 2014, pp. 36–41, 2014.
[36] T. Kumar and K. Verma, “A Theory Based on Conversion of RGB image to Gray
image,” Int. J. Comput. Appl., vol. 7, no. 2, pp. 5–12, 2010.