SYSTEM REQUIREMENT SPECIFICATION

For

Facial Emotion Recognition using Deep Learning

Prepared By

Anagha Dalvi 154012

Kajal Gaikwad 154018

Yamini Shewale 154050

ABSTRACT:
Trying to interpret a person's emotional state in a nonverbal form, usually requires decoding his/hers facial
expression. Many times, body languages and especially facial expressions, tell us more than words about one's
state of mind.
For this project we have performed an experiment which serves multiple purposes:

1. Finding out, once and for all, who "reads" facial expressions better- Men or Women, and if so,
suggesting an answer for the question- why do those differences exist?
2. Revealing special features for recognizing classically defined facial expressions and answering the
question- which facial cues help us the most decipher facial expressions?
3. If possible implement the any application by using the facial application and its implementation which
will help technology sector which will dealing with human expressions.

Moreover, I will try to justify those features from an evolutionary point of view

Key Words: Artificial Neural Network, Emotion, Image Processing, Facial Expression.

PROBLEM STAEMENT:
The scientific study have been made by many organizations on human sociology identifies the real importance
of the human emotion in each and every sector. So if we understand the human emotions automatically it will
help us to process more accurately and efficiently. Understanding the human facial expressions and the study of
expressions has many aspects, from computer analysis, emotion recognition, lie detectors, airport security,
nonverbal communication and even the role of expressions in art. Improving the skills of reading expressions is
an important step towards successful relations.
SYSTEM OVERVIEW:

Project Prospective:
Admin:

Artificial Intelligence is the key feature in the administrator part. This will manage dataset using the clustering
base algorithms and apply them to the system overflow. System administrator is the key player in this system;
Admin is responsible for building the system knowledge base and also evaluates the images and identify the solution with
proper emotions.

User:

Client users of the system which will upload the images in the system from his client portal by authenticate
login process and retrieves the results for the human emotion.

Project Features:
A common assumption is that facial expressions initially served a functional role and not a communicative one.
I will try to justify each one of the seven classical expressions with its functional initially role:

1. Anger: involves three main features- teeth revealing, eyebrows down and inner side tightening,
squinting eyes. The function is clear- preparing for attack. The teeth are ready to bite and threaten
enemies, eyes and eyebrows squinting to protect the eyes, but not closing entirely in order to see the
enemy.
2. Disgust: involves wrinkled nose and mouth. Sometimes even involves tongue coming out. This
expression mimics a person that tasted bad food and wants to spit it out, or smelling foul smell.
3. Fear: involves widened eyes and sometimes open mouth. The function- opening the eyes so wide is
suppose to help increasing the visual field (though studies show that it doesn't actually do so) and the
fast eye movement, which can assist finding threats. Opening the mouth enables to breath quietly and by
that not being revealed by the enemy.
4. Surprise: very similar to the expression of fear. Maybe because a surprising situation can frighten us for
a brief moment, and then it depends whether the surprise is a good or a bad one. Therefore the function
is similar.
5. Sadness: involves a slight pulling down of lip corners, inner side of eyebrows is rising. Darwin
explained this expression by suppressing the will to cry. The control over the upper lip is greater than
the control over the lower lip, and so the lower lip drops. When a person screams during a cry, the eyes
are closed in order to protect them from blood pressure that accumulates in the face. So, when we have
the urge to cry and we want to stop it, the eyebrows are rising to prevent the eyes from closing.
6. Happiness: usually involves a smile- both corner of the mouth rising, the eyes are squinting and
wrinkles appear at eyes corners. The initial functional role of the smile, which represents happiness,
remains a mystery.

USER CLASS AND CHARACTRISTICS:

Admin:

Artificial Intelligence is the key feature in the administrator part. This will manage dataset using the clustering
base algorithms and apply them to the system overflow. System administrator is the key player in this system;
Admin is responsible for building the system knowledge base and also evaluates the images and identify the solution with
proper emotions.

User:

Client users of the system which will upload the images in the system from his client portal by authenticate
login process and retrieves the results for the human emotion.

Clients will executes following functionality by using client application

Client login to the system by using username password

Client upload the image or auto detect the facial image.

By using the facial image system can automatically identifies the user emotion and result displays on the screen.

ADVANTAGES:
 As compare to the existing traditional methods this system have Lower complexity.
 Less computer demanding.
 This system is Precise due to high end algorithms.
 Versatile- system can manage to process any facial image which will make it versatile.
 Extensible.
DISADVANTAGES:
 More computer processing required
 More complexity due to complicated system architecture and complex algorithms.
 Difficult to extend with more emotions.
 Difficult to generalize to new data

SYSTEM ASSUMPTION
It's no secret that women considered to be more intuitive than men. More often, women considered to be more
compassioned and emphatic to their surroundings. Therefore, the gift of interpreting facial expressions is related
usually to women. I believe that my experiment results will support this assumption.
SYSTEM APPLICATION:

 This is a web base application which will implement to identify the human emotions which will help the
organization to identify the human emotions for work distributions and performance analysis.
 This application uses the Artificial Neural Network with knowledge base system and gives great
freedom to administrator of the system to analyze the proper study of human emotions in various sectors
which will help to identify the effects of human emotions on their work and on their surrounding
environment.
1. Introduction:

Trying to interpret a person's emotional state in a nonverbal form, usually requires decoding his/hers facial
expression. Many times, body languages and especially facial expressions, tell us more than words about one's
state of mind.
For this project I have performed an experiment which serves multiple purposes:

4. Finding out, once and for all, who "reads" facial expressions better- Men or Women, and if so,
suggesting an answer for the question- why do those differences exist?
5. Revealing special features for recognizing classically defined facial expressions and answering the
question- which facial cues help us the most decipher facial expressions?

Moreover, I will try to justify those features from an evolutionary point of view

The Importance of Facial Recognition

Understanding the human facial expressions and the study of expressions has many aspects, from computer
analysis, emotion recognition, lie detectors, airport security, nonverbal communication and even the role of
expressions in art.
Improving the skills of reading expressions is an important step towards successful relations.

Expressions and Emotions

A facial expression is a gesture executed with the facial muscles, which convey the emotional state of the
subject to observers. An expression sends a message about a person's internal feeling. In Hebrew, the word for
"face"- ‫פָּנִ ים‬, has the same letters as the word represents "within" or "inside"- ‫פְּ נִ ים‬. That similarity implies about
the facial expression most important role- being a channel of nonverbal communication.
Facial expressions are a primary means of conveying nonverbal information among humans, though many
animal species display facial expressions too.
Although human developed a very wide range and powerful of verbal languages, facial expression role in
interactions remains essential, and sometimes even critical.

Expressions and emotions go hand in hand, i.e. special combinations of face muscular actions reflect a
particular emotion. For certain emotions, it is very hard, and maybe even impossible, to avoid it's fitting facial
expression.
For example, a person who is trying to ignore his boss's annoying offensive comment by keeping a neutral
expression might nevertheless show a brief expression of anger. This phenomenon of a brief, involuntary facial
expression shown on the face of humans according to emotions experienced is called 'micro expression'.

Micro expressions express the seven universal emotions: happiness, sadness, anger, surprise, contempt, fear
and disgust. However, Paul Ekman, a Jewish American psychologist who was a pioneer in the study of
emotions and their relation to facial expressions, expanded the list of classical emotions. Ekman has added to
the list of emotions nine more: amusement, shame, embarrassment, excitement, pride, guilt, relief,
satisfaction and pleasure.
Micro expression is lasting only 1/25-1/15 of a second. Nonetheless, capturing it can illuminate one's real
feelings, whether he wants it or not. That is exactly what Paul Ekman did.
Back in the 80's, Ekman was already known as a specialist for study of facial expressions, when approached by
a psychiatrist, asking if Ekman has the ability to detect liars. The psychiatrist wanted to detect if a patient is
lying by threatening to suicide. Ekman watched a tape of a patient over and over again, looking for a clue until
he found a split second of desperation, meaning that the patient's threat wasn't empty. Since then, Ekman have
found those critical split seconds in almost every liar's documentation. The leading character in the TV series
"Lie to me" is based on Paul Ekman himself, the man who dedicated his life to read people's expressions- the
"human polygraph".

The research of facial expressions and emotions began many years before Ekman's work. Charles Darwin
published his book, called "The Expression of the Emotions in Man and Animals" in 1872. This book was
dedicated to nonverbal patterns in humans and animals and to the source of expressions.
Darwin's two former books- "The Descent of Man, and Selection in Relation to Sex" and "On the Origin of
Species" represented the idea that man did not came into existence in his present condition, but in a gradual
process- Evolution.
This was, of course, a revolutionary theory since in the middle of the 19th century no one believed that man and
animal "obey to the same rules of nature".
Darwin's work attempted to find parallels between behaviors and expressions in animals and humans. Ekman's
work supports Darwin's theory about universality of facial expressions, even across cultures.
The main idea of "The Expression of the Emotions in Man and Animals" is that the source of nonverbal
expressions of man and animals is functional, and not communicative, as we may have thought. This means
that facial expressions creation was not for communication purposes, but for something else.
An important observation was that individuals who were born blind had similar facial expressions to individuals
who were born with the ability to see.
This observation was intended to contradict Sir Charles Bell's idea (a Scottish surgeon,
anatomist, neurologist and philosophical theologian, who influenced Darwin's work), who claimed that human
facial muscles were created to provide humans the unique option to express emotions, meaning, for
communicational reasons.

According to Darwin, there are three "chief principles", which are three general principles of expression:

1. The first one is called "principle of serviceable habits". He described it as a habit that was reinforced at
the beginning and then inherited by offspring.
For example: he noticed a serviceable habit of raising the eyebrows in order to increase the vision field.
He connected it to a person who is trying to remember something, while performing those actions, as
though he could "see" what he is trying to remember.
2. The second principle is called "antithesis". Darwin suggested that some actions or habits might not be
serviceable themselves, but carried out only because they are opposite in nature to a serviceable habit.
I have found this principle very interesting, and I will go into more detail later on.

3. The third principle is called "The principle of actions due to the constitution of the Nervous System".
This principle is independent from will or a certain extent of habit. For example: Darwin noticed that
animals rarely make noises, but in special circumstances, like fear or pain they response by making
involuntary noises.

1.1 Purposes:
Understanding the human facial expressions and the study of expressions has many aspects, from computer
analysis, emotion recognition, lie detectors, airport security, nonverbal communication and even the role of
expressions in art.
Improving the skills of reading expressions is an important step towards successful relations.

1.3 Intended audience:

A common assumption is that facial expressions initially served a functional role and not a communicative one.
I will try to justify each one of the seven classical expressions with its functional initially role:

7. Anger: involves three main features- teeth revealing, eyebrows down and inner
side tightening, squinting eyes. The function is clear- preparing for attack. The
teeth are ready to bite and threaten enemies, eyes and eyebrows squinting to
protect the eyes, but not closing entirely in order to see the enemy.

8. Disgust: involves wrinkled nose and mouth. Sometimes even involves tongue
coming out.
This expression mimics a person that tasted bad food and wants to spit it out, or
smelling foul smell.

9. Fear: involves widened eyes and sometimes open mouth.

The function- opening the eyes so wide is suppose to help increasing the visual
field (though studies show that it doesn't actually do so) and the fast eye
movement, which can assist finding threats. Opening the mouth enables to breath
quietly and by that not being revealed by the enemy.

10. Surprise: very similar to the expression of fear.

Maybe because a surprising situation can frighten us for a brief moment, and then it
depends whether the surprise is a good or a bad one. Therefore the function is
similar.

11. Sadness: involves a slight pulling down of lip corners, inner side of eyebrows is
rising. Darwin explained this expression by suppressing the will to cry. The control
over the upper lip is greater than the control over the lower lip, and so the lower lip
drops. When a person screams during a cry, the eyes are closed in order to protect
them from blood pressure that accumulates in the face. So, when we have the urge to
cry and we want to stop it, the eyebrows are rising to prevent the eyes from closing.
 12. Contempt: involves lip corner to rise only on one side of the face. Sometimes
only one eyebrow rises. This expression might look like half surprise, half
happiness.
This can imply the person who receives this look that we are surprised by what
he said or did (not in a good way) and that we are amused by it. This is obviously
an offensive expression that leaves the impression that a person is superior to
another person.

13. Happiness: usually involves a smile- both corner of the mouth rising, the eyes are
squinting and wrinkles appear at eyes corners. The initial functional role of the
smile, which represents happiness, remains a mystery. Some biologists believe that
smile was initially a sign of fear. Monkeys and apes clenched teeth in order to
show predators that they are harmless. A smile encourages the brain to release
endorphins that assist lessening pain and resemble a feeling of well being. Those
good feeling that one smile can produce can help dealing with the fear. A smile can
also produce positive feelings for someone who is witness to the smile, and might
even get him to smile too.

Newborn babies have been observed to smile involuntarily, or without any external stimuli while they
are sleeping. A baby's smile helps his parents to connect with him and get attached to him. It makes
sense that for evolutionary reasons, an involuntary smile of a baby helps creating positive feelings for
the parents, so they wouldn't abandon their offspring.

1.4 Project scope:

A facial expression is a gesture executed with the facial muscles, which convey the emotional state of the
subject to observers. An expression sends a message about a person's internal feeling. In Hebrew, the word for
"face” has the same letters as the word represents "within" or "inside". That similarity implies about the facial
expression most important role- being a channel of nonverbal communication.
Facial expressions are a primary means of conveying nonverbal information among humans, though many
animal species display facial expressions too.
Although human developed a very wide range and powerful of verbal languages, facial expression role in
interactions remains essential, and sometimes even critical.

Expressions and emotions go hand in hand, i.e. special combinations of face muscular actions reflect a
particular emotion. For certain emotions, it is very hard, and maybe even impossible, to avoid it's fitting facial
expression.
For example, a person who is trying to ignore his boss's annoying offensive comment by keeping a neutral
expression might nevertheless show a brief expression of anger. This phenomenon of a brief, involuntary facial
expression shown on the face of humans according to emotions experienced is called 'micro expression'.

Micro expressions express the seven universal emotions: happiness, sadness, anger, surprise, contempt,
fear and disgust. However, Paul Ekman, a Jewish American psychologist who was a pioneer in the study of
emotions and their relation to facial expressions, expanded the list of classical emotions. Ekman has added
 to the list of emotions nine more: amusement, shame, embarrassment, excitement, pride, guilt, relief,
satisfaction and pleasure.

1.5 Reference:
[1] Geoffrey E. Hinton, Yee-Whye Teh and Simon Osindero, A Fast Learning Algorithm for Deep Belief Nets.
Neural Computation, pages 1527-1554, Volume 18, 2008.

[2] Susskind, J.M. and Hinton, G.E. and Movellan, J.R. and Anderson, A.K., Generating facial expressions with
deep belief nets, Affective Computing, Emotion Modelling, Synthesis and Recognition, pages 421-440,2009

[3] Michael J. Lyons, Shigeru Akamatsu, Miyuki Kamachi & Jiro Gyoba , Coding Facial Expressions with
Gabor Wavelets,Proceedings, Third IEEE International Conference on Automatic Face and Gesture
Recognition,pp 200-205, 1-19.April 14-16 1998

[4] Geoffrey E. Hinton (2010). A Practical Guide to Training Restricted Boltzmann Machines,Technical
Report,Volume 1

[5] Andrew Ng. (2010). Sparse autoencoder(lecture notes). [6] Honglak Lee, Chaitanya Ekanadham, Andrew Y.
Ng, Sparse deep belief net model for visual area V2NIPS,2007

2 Overview:
Facial expressions are important cues for non verbal communication among human beings. This is only possible because
humans are able to recognize emotions quite accurately and efficiently. An automatic facial emotion recognition system is
an important component in human machine interaction. Apart from the commercial uses of an automatic facial emotion
recognition system it might be useful to incorporate some cues from the biological system in the model and use the model
to develop further insights into the cognitive processing of our brain.

One problem with ad hoc feature extraction schemes is that we need to design separate feature extraction mechanism foe
each visual task to be performed. Moreover, it is known that only some of the filters in the retina are hardcoded and the
other units in the V1,V2 and higher areas of visual processing are learned. Hubel and Wiesel showed that irreversible
damage was produced in kittens by sufficient visual deprivation during the so called ”critical period”. Therefore, it makes
much more sense to have generic scheme for learning what transformations in the input space may lead to good features
for performing a particular task.

2.1 Project Perspective:

State of the art approaches in facial emotion recognition use Active Appearance Models (AAMs), FACS labels or some
other sophisticated feature extraction scheme. AAMs can be learned from a set of training images and can be fitted on a
new face to generate the landmark positions which can further be used to design features. Thus, in an automatic setting
either the availability of landmark point on face images is assumed or can be obtained by fitting the model. FACS labels
attempt to decompose human emotions in terms of Action Units(AUs) which correspond to specific muscle movements.
FACS coding system is used in psychology and animation to classify facial expressions in a consistent and systematic
manner. But as of now FACS labels can only be given by experts or trained individuals.
2.2 Project function:

The antithesis phenomenon refers to the way that some muscle movements represent an emotion, and the
opposite muscle movements represent the opposite emotion.
An impressive explanation for the facial expression represents 'helplessness' can be done using antithesis:
Helplessness body gesture involves hands spreading to the sides, fingers spreading and shoulders shrugging. It's
facial expressions involves pulling down the bottom lip and raising eyebrows, like you can see in the followed
image:

Darwin explained the features of this expression using the antithesis principle. He discovered that all of those
movements opposing to the movements of a man who is ready to face something. The movements of a person
who is preparing himself for something will look like that: closed hands and fingers (as if he is preparing for a
fight, for example), hands close to the body for protection and the neck is raised and tight. At a helplessness
situation the shrugging of the shoulders releases the neck.
As for the face: eyebrows are low (like in a mode of attack or firmness), upper lip might reveal teeth.

The functional source of the antithesis can be explained with the investigation of muscles, and to be precise- the
antagonist's muscles. Every muscle has an antagonist muscle that performs the opposite movement. Spreading
fingers is a movement done by some muscles, and closing the fingers is done by the antagonist muscles. For
some expressions we can't always tell just by looking at it, what is the opposite expression, but if we'll look at
the muscles involving in the process then it becomes very clear.

2.3 User class and characteristic:

2.4 Operating environment:

Operating environment for the online auction system is as list below:

Distributed database client/server system operating system windows.

Platform: Java, HTML, JAVA

2.5 Design and implementation constraints:

2.6 Dependency:
The system can be used only if the user login with correct password and use rid. Further transection can be
carried on only with the permission of the adminstrator.
SOFTWARE REQUIREMENT:
 Windows xp or 7 and above
 MySQL Server
 JAVA

HARDWARE REQUIREMENT:

 Processor – Intel i3
 Hard Disk – Minimum 5 GB
 Memory – 1GB RAM

3. External interface:
The System is accessible for any operating system using a web browser and connection to the Web server
running the client software no special hardware is required by the end user the client browser must be
compatible communication between the user and server will be through HTTP communication using Tcp ip
ports, an error occur during a request.the user should receive a clear error message.

3.1 User interface

This interface will be in the form of Web app.it is designed to the functional and minimal in its styling. All
product will be displayed in image and descriptions format. Setup the page layout and add minimal styling to
make the interface user friendly

3.2 Software Interface:

The server will be hosted using apache webserver.it will also have a MYSQL relational database.the main
backend processing done using java including connecting to and accessing the database and processing requets.

3.3 Communication Interface:

The main communication protocol will be hyper text transfer protocol (HTTP). This will be used to transfer
information back And forth form to client to server.HTTP and GET to use the send the information securely over the
browser.

5.1 Performance Requirement:

Physical machine to be used in the system need to have the internet access in order to connect to the database.
This software will not assume that code scanner hardware are available on the machine and so the USER_ID
inputwill be done via keyboard and mouse. user need internet access on their device as well since the all the
data are stored on the on the server database which the software will need to connet it.

5.2 Security Requirements:

The online system will use the security for the system administrator.

Login_Id and passwordis associated with the system administrator to provide security over the system.

No detail of the competitor bidder will be shown as it can be case of fraud.

5.3 Software quality attributes:

1. The detail of the user kept secret.

2.No case of fraud is possible.

3.AVAILABILITY: The user need to upload the facial images in the system according to the facial expression the user
emotion will be identified and notified to the client user.

4.CORRECTNESS: The correct Human emotion detection is the main purpose of the project hence According to the
facial expression the user emotion will be identified and notified to the client user.

5.MAINTAINABILITY: The administrators can maintain correct dataset for the user images and the emotions in
accordance.

5.4 Other requirements:

a) Business rules: This is a complex system with lot of interfaces, hence administration has to be maintain the
correct database for each and every image and its exact relation to the emotion with reference to the facial
expressions.

Approach and Method

Experiment Description
My experiment has been composed from 2 sections:
First, a series of 35 full facial images, each representing one of the seven different facial expressions, is
presented to each participant. Each picture appears for 4 seconds only, followed by a black screen, in order to
prevent the participant adapt to the image. The participant labels each picture as one of the seven given facial
expressions and fills in a given form, made for this purpose, during the "black screen".
The participant also needs to fill in which 1-3 facial features assisted him/her to classify the image as is.
Second, a series of 21 facial features images is presented to each participant, again for 4 seconds only. The
participant classifies each feature to a facial expression and fills in the form, during the "black screen". The
purpose of the second part was to see if a person can identify expressions better or worse using only some facial
parts and also to see if the time for recognition changes.
20 men and 20 women took part in my experiment.
My Assumption
It's no secret that women considered to be more intuitive than men.
More often, women considered to be more compassioned and emphatic to their surroundings. Therefore, the gift
of interpreting facial expressions is related usually to women. I believe that my experiment results will support
this assumption.

Experiment Results
I have analyzed each of the seven facial expressions, and compared the success percentage of men and women
in each part of the experiment.

Happy
1.00 98% 96% 100% 100%

0.80

0.60

0.40

0.20

0.00

As we can see in the graph, women and men identified happiness in similar percentages. The most interesting
part, in my opinion, is that happiness was identified in almost 100% success. My assumption is that happiness,
being a positive emotion, is a mood that people wants to be around. Happy people project their feelings to
others and help to create good vibes in their surroundings. It is important to recognize easily this kind of
emotion because it is helpful to be around. Maybe this is the reason for easy identification of happiness.

Sad
1.00

0.80 82% 78%

73% 72%
0.60

0.40

0.20

0.00
The graph shows that though women recognized sadness better than men in full facial images, men recognized
it better in the facial features images.

Contempt
1.00

0.80
69%
0.60 63% 60%
54%
0.40

0.20

0.00

This expression was difficult to recognize for both men and women. In the first part of the experiment, most of
the mistakes were labeling sadness instead of contempt.
On the other hand, in the second part, most of the mistakes were labeling happiness instead of contempt. I think
that the confusion in the second part came from the mouth shape for contempt, which involves lip corner to rise
on only one side of the face. Most of the people indicated that the eyebrows, along with the lips, helped them to
recognize the expression.
I believe that the shape of the eyebrows got people to confuse contempt with sadness in the first part. This
expression is a tricky one, and I have expected confusions in its identification.

Surprise
1.00
93%
88% 86%
0.80
72%
0.60

0.40

0.20

0.00
 Surprisingly, this expression was recognized better by men.

Angry Disgust
1.00 1.00
87% 89% 92% 91%
80% 84%
0.80 76% 0.80 74%
0.60 0.60

0.40 0.40

0.20 0.20

0.00 0.00

Fear
1.00

0.80

0.60 56% 53%

0.40 38%
24%
0.20

0.00

Those three expressions- Fear, Disgust and Anger, were recognized better by women, and we can see that most
of the times the difference in percentage of success between men and women are significant.
An interesting question that should be asked is- why are those specific emotions were recognized better by
women?
From an evolutionary point of view, evolutionary psychologists have suggested that females, due to their role as
primary caretakers, are "programmed" to accurately decode and detect distress in preverbal infants or
threatening signals from other adults to enhance their chances to survive. Fear, anger and disgust are indeed
situations of distress.

The most common two special features that helped the participants to decode the expression and the emotion
behind it were the lips and the eyebrows. I have also found out that most of the times, in the second part of the
experiment, the recognition took less than 4 seconds. This means that there are special features for each
emotion, and if we focus on that features alone we will be able to decode the expression faster.
 Conclusions
I have analyzed men and women results for all the seven emotions and those are the final results:

The final result shows that women do decode expression better in both parts of the experiment, though the
difference between sexes was very small in the second part of the experiment.
Are women natured or nurtured to be different? That is still a mystery and an interesting lead to follow.