Modules of Project:

The project is primarily divided into two modules which are connected in such a
way that one stores the details for the training phase and second one handles
the video rendering and face expression capture for prediction in realtime. The
other modules used for it are a part of python libraries and they compliment
the process.The other modules which are equal important are Tenserflow
,Keras , Pandas , Numpy and some other small packages as well.

First Module:
The first module consist of the code that will create the CNN model and will get
the data for training purpose . This module will get the model trained and will
store it in some format like either in pickle or json . This trained model will
further be used for the prediction as we won’t require to train the model again
and again and can do the realtime prediction .

In this module we will use the pandas ,numpy and Keras for the complete
implementation of CNN as keras will be used to implement the Neural Network
and pandas and numpy will be used to handle the data for the training
purpose.In this module we will also use a json handling logic to store the
trained model in json format or pickle format if we use pickle for the model
storage.

Keras:
Keras is a high-level neural networks API, written in Python and capable of
running on top of TensorFlow, CNTK, or Theano. It was developed with a focus
on enabling fast experimentation. It is the library which helps to implement the
neural network in python and also gives the ways to modify the network
accordingly and refine for a better efficiency .There are two main types of
models available in Keras: The Sequential Model, and The Model Class used
with the functional API. In this project we are panning to use sequential model
for the prediction as i have already done some projects in it and they give a
better efficiency as cpmapre to second model.

Pandas :
It is used to get the data from csv file and modify , cut and get the required
data for the use in model . It is used to create dataframe during the data is
picked or modified.

Numpy:
It is used to perform the calculations in a very easier way and with a great
efficiency. It will be required in project to modify the data and covert to the
required format for network training.

Module 2:
The second module handles the part of project which will capture the realtime
video and will give the output on the basis of that video. In this module we
used openCV , pandas , numpy and Keras. This will be the main file of the
project which will be executed every time we will execute the process of mood
detection. This module will require the data that is stored in json format for
further prediction and hence the keras in it is helpful to again change that
model from json to working model . The keras pics the model from json format
and let the module to perform prediction with the help of it.

OpenCV:
It is a python based computer vision library which helps to handle the images
and video data . In this project it will be used to handle the video data and
capture the input data for the prediction and will also be required to show the
realtime output of it.

Keras :
keras will be used to get the trained model . As we will store the trained model
in a json file and keras will revert it back to actual model which will be used for
the prediction.

Numpy:
numpy will be used to handle the input data and modify it accordingly . We will
use this to change the given data into the similar format as the training data
was so as for better prediction .

Process :
In this project as i have already mentioned that there are two primary modules which integrates to
form the complete project and make it work properly. Hence these modules perform a tons of
process in them and these process include model formation ,prediction etc. So to detail the primary
processes of the project we will do it in two major parts based on two modules. So starting with the
details of process:

Module 1 Process detail:

In this module we will actually do the whole process of making a neural network(CNN) and
training it with the help of data provided to us. In this we will begin with the importing part to get
the necessary libraries we will require for our work. We will mainly import libraries like os, sys,
numpy , pandas, keras and some other methods of keras .

After this we will firstly get the data which we got from the kaggle and will modify it as per our
requirement using numpy. We will pick the data using pandas and modify using numpy . Then we
will divide the data for dependent and independent variables for training the network . This data
will be divident on the basis of some specific formulae so that the whole data could be completely
randomised and should give good efficiency .

After cleaning and preparing the data for the next step we will start preparing the model for
prediction. To start preparing we will use the keras to form the convolutional layers and planning to
have 3 layers at most. After these layers are formed we now train the network using the data we
have preproccessed.
And after training the network we will store the trained model into some json or pickle format
which will be required in furture for prediction.

Module 2 Process Detail:

This model will handle the complete the process to perform the prediction task .In this module we
will use the stored trained model to perform the task . We will import the necessary libraries along
with Keras to startwith. Firstly we will use the keras to get the trained model from the json format to
actual format of model which will be used for the training purpose .

We will use the model from json method to get the model and then we will use the opencv tool to
get the live video capture and this video will help to perform the emotion detection process from
form the same captured video in live mode.Now when we will capture the video firstly we will
cross verify if video capturing started & when it will ,we will do the proccessing of that video to tell
the output of it.

Now we will preprocess the frames such that we will detect only the face part of image/frame and
will covert it into the specific color and pixel . Now we will use the inbuilt features of openCV to
perform this modification of images and covert to the similar pixels as of the trained dataset .

Now these images will be coverted to pixels by some other methods which will be used to be as
input data.Now the prediction will be checked that which category it falls as in the list of all those
6-7 emotions. Now we will provide the output on the same frame as in we are getting the video and
for this we will again resize the image to fit in a frame and will give the output of emotion on same
frame at some place at fixed location.
Dataset Description:
The emotion detection process requires a huge amount of data so that the
model can be trained on this data and will provide a better and precised
output. This data works as a knowledge base for the model and lets it to
understand the pattern of face. Hence to get a perfect data we surfed the web
and got data at various places but as we all know there is a hub of everything
and hence the hub for good quality dataset is Kaggle ,the place which have
nearly all good and latest dataset for ML and AI projects.

Hence we searched for the dataset there as well and got the dataset for this
particular project as there was a competition at the same website which was
held to build some this type of project . The link to the dataset is :
https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-
challenge/data

This dataset consist of three columns and a total of 35888 rows which consist
of emotion , pixel information ,and usage details. These are given in details as
below:
Emotion Information:In the very first column the information about the emotion
is given in form of values from 0-6 (a total of 7 emotions) . The emotions are
given in a series from 0-6 as anger, disgust, fear, happy, sad, surprise,
neutral .Facial expression presents key mechanism to describe human emotion.
From starting to end of the day human changes plenty of emotions, it may be
because of their mental or physical circumstances.Basic facial features are
eyebrow, mouth, nose & eyes. As the details of six of them are as below as in
neutral the things are completely taken as base /initial hence its details are not
mentioned there.
Pixel Information:

In this dataset the details of images is provided in forms of pixel information in

48*48 pixel matrix basis. The second column of it stores this details of images
in pixel forms .The image is hence easy to handle as it is already in pixel form
and we didn’t have to modify it further.
In these pictures all the details are in same size and format so it is really an
easy task to feed them into the network for training purpose. The shot of the
datset is as below:

The third details given there is for the specific purpose of competition to decide
how much is to be used for what pupose as like training ,testing etc. We will
use either the train test split to decide the data for training and testing pupose
or we will use that thirsd given column to perform the same action.
Hence the total data we have is 3 columns and 35888 rows and we will use
more than half of data for training purpose and the rest for testing to find
efficiency.