Sei sulla pagina 1di 24

DROWSINESS DETECTION FOR CAR ASSISTED DRIVER SYSTEM USING

IMAGE PROCESSING ANALYSIS INTERFACING WITH HARDWARE

HUONG NICE QUAN

This thesis is submitted as partial fulfillment of the requirements for the award of the
degree of Bachelor of Electrical Engineering (Electronics)

Faculty of Electrical & Electronics Engineering


Universiti Malaysia Pahang

NOVEMBER, 2010

ii

All the trademark and copyrights use herein are property of their respective owner.
References of information from other sources are quoted accordingly; otherwise the
information presented in this report is solely work of the author.

Signature

: ____________________________

Author

: HUONG NICE QUAN

Date

: 24 NOVEMBER 2010

iii

To my beloved father, mother, brothers and sister


To my respectful supervisor Dr. Kamarul

iv

ACKNOWLEDGEMENT

I would like to take this opportunity to show my gracefulness to my


respectful supervisor, Dr. Kamarul Hawari bin Ghazali for his precious guides
throughout this project. He spent so much time with me to discuss and help with this
project. His ideas, helps, and guides keep my passion throughout these two semesters.

I would like to thank to all UMP lecturers especially and electrical


technicians whom had helped directly or indirectly in what so ever manner thus
making this project a reality.

My deepest thanks to my dearest family members who are always support me


in every aspect. Their encouragement and support gave me the strength to overcome
problems throughout this whole project.

Again, to any parties that gave great cooperation and helping-hands will be
always appreciated.

ABSTRACT

The purpose of this study is to detect drowsiness in drivers to prevent


accidents and to improve safety on the highways. A method for detecting
drowsiness/sleepiness in drivers is developed by using a camera that point directly
towards the drivers face and capture for the video. Once the video is captured,
monitoring the face region and eyes in order to detect drowsy/fatigue. The system
able to monitoring eyes and determines whether the eyes are in an open position or
closed state. In such a case when drowsy is detected, a warning signal is issued to
alert the driver. It can determine a time proportion of eye closure as the proportion of
a time interval that the eye is in the closed position. If the drivers eyes are closed
cumulatively more than a standard value, the system draws the conclusion that the
driver is falling asleep, and then it will activate an alarm sound to alert the driver.

vi

ABSTRAK

Tujuan pengkajian ini adalah untuk mengesan pemandu yang menghadapi


masalah mengantuk semasa memandu dan menggelakkan kemalangan serta
meningkatkan keselamatan di jalan raya. Kaedah yang digunakan untuk mengesan
kantuk/ngantuk terhadap pemandu dengan menggunakan kamera mengarah langsung
ke arah wajah pemandu untuk dan merakam untuk video. Setelah video dirakam,
pematauan terhadap wajah dan mata untuk mengesan masalah mengantuk/keletihan.
Sistem ini mampu memantaukan dan menentukan mata dalam keadaan terbuka atau
keadaan tertutup. Dalam kes ketika mengantuk dikesan, isyarat amaran dikeluarkan
untuk memberi amaran kepada pemandu. Sistem ini dapat menentukan perkadaran
masa penutupan masa sebagai perkadaran selang masa yang mata dalam kedudukan
tertutup. Sekiranya mata pemandu tertutup secara kumulatif lebih daripada nilai
standard, sistem tersebut akan membuat kesimpulan bahawa pemandu dalam
keadaan mengantuk, dan kemudian akan mengaktifkan suara penggera untuk
memberi isyarat dan amaran kepada pemandu.

vii

TABLE OF CONTENTS

CHAPTER

TITLE

PAGE

DECLARATION

ii

DEDICATION

iii

ACKNOWLEDGEMENT

iv

ABSTRACT

ABSTRAK

vi

TABLE OF CONTENTS

vii

LIST OF TABLES

xi

LIST OF FIGURES

xii

LIST OF ABBREVIATIONS

xiv

INTRODUCTION

1.1

Introduction to the Project

1.2

Problem Statement

1.3

Objective

1.4

Scope of Project

1.5

Thesis Overview

viii

TABLE OF CONTENTS

CHAPTER

TITLE

LITERATURE REVIEW

2.1

Face Detection Technologies

2.2

Efficient Eye States Detection in


Real-Time for Drowsy Detection

2.3

2.4

2.5

Drowsiness Detection Based on Brightness


and Numeral Features of Eye Image

11

Drowsy Detection and Alarming System


(DroDeASys)

14

2.4.1

Pre-processing

16

2.4.2

Processing

16

2.4.3

Post Processing

17

Drowsiness Detection System Using


Electrooculogram (EOG)

PAGE

17

METHODOLOGY

21

3.1

Introduction

21

3.2

Image Acquisition

21

3.2.1

Illumination

22

3.2.2

Camera

22

3.3

Face Detection and Eye Detection Function

23

3.4

Determining the State of the Eyes

24

3.5

Drowsy Decision

26

ix

TABLE OF CONTENTS

CHAPTER

TITLE

PAGE

HARDWARE AND SOFTWARE


IMPLEMENTATION

28

4.1

Camera Hardware

28

4.2

Different Between CMOS


And CCD Technology

31

4.2.1

Feature and Performance Comparison

33

4.2.2

CMOS Developments Winding Path

34

4.3

Image Processing

35

4.4

MATLAB History

36

4.5

MATLAB Introduction

36

4.6

MATLAB - Image Processing Toolbox

37

RESULTS AND DISCUSSION

39

5.1

Simulation Results

39

5.2

Analysis

43

5.2.1

43

Handling videos in MATLAB

5.2.1.1 Loading Video Files - Read Audio/Video


Interleaved (AVI) File

43

5.2.1.2 Editing Frames

44

5.2.2

44

Pre-Processing

TABLE OF CONTENTS

CHAPTER

TITLE

5.3

PAGE

5.2.3

Processing

45

5.2.4

Post-Processing

53

Discussion

55

CONCLUSION

56

6.1

Achievement

56

6.2

Limitations

57

6.3

Future Recommendation

57

REFERENCES

59

APPENDICES

61

xi

LIST OF TABLES

TABLE NO.

TITLE

PAGE

1.1

Gantt chart PSM 1

1.2

Gantt chart PSM 2

2.3.1

Accuracy of open, semi-open, closed eye


determination on database

10

xii

LIST OF FIGURES

FIGURES NO.

TITLE

PAGE

2.1

Overview of drowsy driving monitoring system

2.2

Examples of effective regions selected by AdaBoost

2.3

PERCLOS measurements for alert and drowsy data

2.4

Flowchart of the system for drowsiness detection

10

2.5

Variance projection curve in vertical


direction of different eye states

2.6

11

Measured PERCLOS parameter for one person in


non-drowsy and drowsy states with warning message 11

2.7

Tested Samples

13

2.8

Block diagram of DroDeASys

14

2.9

Electrode placement positions for EOG


measurement

16

2.10

Instrument setup during data collection

17

2.11

Summary of the EOG process algorithm

17

3.1

Process flow of image binarization

21

3.2

Output comparison between eyes opening state

22

3.3

Flowchart of the proposed system for


drowsiness detection

5.1

24

Demonstration of first step in the process


video playback

36

5.2

Demonstration of binarizing the image

37

5.3

Demonstration of the location of eyes

37

5.4

Result show of drowsy detected

38

5.5

Result displays of drowsy detected


in command window

38

xiii

LIST OF FIGURES

FIGURES NO.

TITLE

5.6

Result show of no drowsy detected

5.7

Result display of normal state

PAGE

39

in command window

39

5.8

Different threshold values for binary image

44

5.9

Eye region cropped

5.10

Pixel values of eye region

46-47
49

xiv

LIST OF ABBREVIATIONS

PERCLOS

Percentage of Eyelid Closure Over the Pupil over time

CMOS

Complimentary Metal-Oxide Semiconductor

CCD

Charge-Coupled Device

AVI

Audio/Video Interleaved

MPEG

Moving Picture Experts Group

MP4

Moving Picture Experts Group Part 14

3GP

Third Generation Partnership Project

PDA

Personal Digital Assistant

JPEG

Joint Photographic Experts Group

TIFF

Tagged Image File Format

PNG

Portable Network Graphics

HDF

Hierarchical Data Format

FITS

Flexible Image Transport System

ASCII

American Standard Code for Information Interchange 2

BIP

Basic Imaging Profile

CHAPTER 1

I TRODUCTIO

1.1

Introduction to the Project

Driving with drowsiness is one of the main causes of traffic accidents. Driver
fatigue is a significant factor in a large number of vehicle accidents. The
development of technologies for detecting or preventing drowsiness at the wheel is a
major challenge in the field of accident avoidance systems. Due to the hazard that
drowsiness presents on the road, methods need to be developed for counteracting its
affects.

There are many technologies for drowsiness detection and can be divided into
three main categories: biological indicators, vehicle behavior, and face analysis [1].
The first type measures biological indicators such as brain waves, heart rate and
pulse rate. These techniques have the best detection accuracy but they require
physical contact with the driver [2]. They are intrusive. Thus, they are not practical.
The second type measures vehicle behaviors such as speed, lateral position and
turning angle. These techniques may be implemented non-intrusively, but they have
several limitations such as the vehicle type, driver experience and driving conditions.
Furthermore, it requires special equipment and can be expensive. The third type is

face analysis. Since the human face is dynamic and has a high degree of variability,
face detection is considered to be a difficult problem in computer vision research. As
one of the salient features of the human face, human eyes play an important role in
face recognition and facial expression analysis. In fact, the eyes can be considered
salient and relatively stable feature on the face in comparison with other facial
features. Therefore, when we detect facial features, it is advantageous to detect eyes
before the detection of other facial features. The position of other facial features can
be estimated using the eye position [3]. In addition, the size, the location and the
image-plane rotation of face in the image can be normalized by only the position of
both eyes.

The aim of this project is to develop a drowsiness detection system. The


vision-based systems have been widely used because of its accuracy and nonintrusiveness. Visual cues such as eye states (i.e. whether they are open or closed)
can typically reflect the drivers level of fatigue. Therefore, an automatic and robust
approach to extract the eye states from input images is very important. The focus will
be placed on designing a system that will accurately monitor the open or closed state
of the drivers eyes. By monitoring the eyes, it is believed that the symptoms of
driver fatigue can be detected early enough to avoid a car accident. Detection of
fatigue involves a sequence of images of a face, and the observation of eye
movements and blink patterns.

This project is focused on the localization of the eyes, which involves looking
at the entire image of the face, and determining the position of the eyes. Once the
position of the eyes is located, the system is designed to determine whether the eyes
are opened or closed, and detect fatigue.

1.2

Problem Statement

Driver drowsiness is a serious hazard in transportation systems. It has been


identified as a direct or contributing cause of road accident [4]. Driver drowsiness is
one of the major causes of road accident. Drowsiness can seriously slow reaction
time, decrease awareness and impair a driver's judgment. It is concluded that driving
while drowsy is similar to driving under the influence of alcohol or drugs [5]. In
industrialized countries, drowsiness has been estimated to be involved in 2% to 23%
of all crashes [6].Systems that detect when drivers are becoming drowsy and sound a
warning promise to be a valuable aid in preventing accidents.

1.3

Objective

The objectives of this project are to develop a drowsiness detection system


that can detect drowsy or fatigue in drivers to prevent accidents and to improve
safety on the roads. This system able accurately monitors the open or closed state of
the drivers eye. When drowsy is detected toward a driver, a warning signal is issued
to alert the driver.

1.4

Scope of Project

The simulation and analysis based on eyes scanning using image processing
technology. This detection involves- observation of eyes that is in open or closed
state and the blinking patterns for a driver. A warning signal will be generated to
trigger a hardware device (alarm) to alert user.

1.5

Thesis Overview

This Drowsiness Detection for Car Assisted Driver System Using Image
Processing Analysis - Interfacing with Hardware final thesis is a combination of five
different chapters. Each of the chapters elaborates details regarding different aspects.
The included aspects are Introduction, Literature Review, Methodology, Hardware
and Software Implementation, Result and Discussion, and Conclusion. Furthermore,
the Gantt charts in table 1.1 and table 1.2 show that the detailed of progress.

Chapter 1: Basic introduction of the this project

Chapter 2: Literature Review for the development of this project


Chapter 3: Method used throughout the development of the whole project

Chapter 4: Hardware & Software Implementation for the project.


Chapter 5: Results and Discussion on the performance of this project.
Chapter 6: Conclusion of this project.

CHAPTER 2

LITERATURE REVIEW

This chapter reviews about the studies that have been done before or during
the development of this project. It is the summary of all related study material and
components required in this research. All ideas and concepts yield are to be
implemented on the research.

2.1

Face Detection Technologies

Due to the human face is dynamic and has a high degree of variability; face
detection is considered to be a complex task in computer vision. Despite its
difficulties, scientists and computer researchers have developed and improved
various face detection techniques.

Face detection is a necessary step in all face processing systems, and its
efficiency influences the overall performance of drowsiness detection systems.
Researchers classified the face detection techniques using the following approaches:

the top down model based approach (search different face model at different scales
level), bottom up feature based approach (searches the image for a set of facial
features), texture based approach (faces are detected by examining the spatial
distribution of gray or colour information), neural network approach (detects faces
by sampling different regions and passing it to neural network), colour based
approach (labels each pixel according to its similarity to skin colour and face shape),
motion based approaches (use image subtraction to extract the moving foreground
from the static background).

Besides, another major classification categorizes the face detection


algorithms into the following approaches: feature-based, image-based, and template
matching. The general classification for face detection algorithms and supported
tools are presented by Hjelm [7] and it can be divided into three categories: feature
based, template matching, and image based.

2.2

Efficient Eye States Detection in Real-Time for Drowsy Detection

A reliable method of eye states detection in real-time for drowsy monitoring


by given a restricted local block of eye regions, the Local Binary Pattern (LBP)
histogram of the block is extracted and each bin of the histogram is treated as a
feature of the eye and followed by an AdaBoost based cascaded classifier is trained
to classify the eye states as open or closed. According to the states of the eye, the
PERCLOS (the percentage of time that an eye is closed in a given period) score is
measured to decide whether the driver is at drowsy state or not.

Figure 2.1: Overview of drowsy driving monitoring system

Figure 2.2: Examples of effective regions selected by AdaBoost

AdaBoost- learning is an algorithm which maximizing classification margin


iteratively.

The face and eye detectors are built based on the standard AdaBoost training
methods combined with Violas cascade approach using haar-like features [8]. This
method of face and eye detection has been proved to be fast and effective enough for
real-time eye states detection system, even under weak or strong light conditions, as
long as the training data include these situations.

Next, the experimental results on eye-state detection are based on the


assumption that eye regions of each frame are all corrected located. By the use of
cascaded AdaBoost for learning effective features from the large feature set and
discard redundant information. It show the effectiveness most of the blocks are
centralized at the regions of pupil, eye corners or eyelids, which are evidently the
distinctive regions for distinguishing open and closed states.

Finally, a decision about drowsiness is made by measuring the PERCLOS


(Percentage of eye closure over time). PERCLOS is the most popular method of
measuring eye blinking because high PERCLOS scores are strongly related to
drowsiness [9]. The time that the eye is closed is continuously accumulated for the
latest 30 seconds in order to acquire the PERCLOS. Fig. 1 is the plots of PERCLOS
measured over 150 seconds. For the alert state, the graph is more stable and the score
is much lower than the drowsy one. When the score exceeds 30%, warning message
is given for the drowsy state by the system.

Figure 2.3: PERCLOS measurements for alert and drowsy data

2.3

Drowsiness Detection Based on Brightness and Numeral Features of Eye


Image

An algorithm for eye state analysis, which incorporates into a four step
system for drowsiness detection: face detection, eye detection, eye state analysis, and
drowsy decision. It requires no training data at any step or special cameras. The
novel eye state analysis algorithm detects open, semi-open, and closed eye during
two steps which is based on brightness and numeral features of the eye image.

Figure 2.4: Flowchart of the system for drowsiness detection

Table 2.1: Accuracy of open, semi-open, closed eye determination on database

Eye State
(eye frame in
special state/ total
eye frame)

Variance
based
algorithm

Eyelids
distance
based
algorithm

The
proposed
algorithm

Accuracy
(%)
96.4

Accuracy
(%)
100

Accuracy (%)

Semi-Open
(375/2250)

100

95.2

95.2

Closed (750/2250)

67.5

72.5

94.7

Open (1125/2250)

96.4