Seminar Paper: Emotion Recognition from Physiological signals

using Bio-sensors
Revathi priya Muthusamy,
University of Fribourg, Switzerland,
Revathipriya.muthusamy@unifr.ch

Abstract:
This paper presents an introduction to various bio-signals used in
Emotion recognition such as Electromyography (EMG)
electrocardiogram (ECG), electrodermal response (EDR), Skin
temperature, and Blood volume pulse (BVP) and respiration
changes. This paper also discusses about various kinds of bio-
sensors used to detect the Physiological signals. In the remaining
of the paper we describe the Multimodal emotion elicitation and
Acquisition of emotion-specific physiological signal database. It
also deals with feature extraction technique based on the Hilbert-
Huang Transform (HHT) method and classification techniques
such as Support Vector Machine. the highest recognition accuracy
is obtained by using the fission based HHT features method with
28 features, 76% of test cases are correctly classified.
Keywords:
Bio-sensors, Multimodal emotion elicitation, Hilbert-Huang
Transform, Support Vector Machine
1. Introduction to Biomedical Signals
Most physiological processes produce signals of several
types:
 Biochemical -in the form of hormones.
 Electrical - in the form of current, and
 Physical - in the form of pressure and temperature. [1]
For example when we get afraid of something our heart
races, our muscles tense, our mouth becomes dry. These changes
are controlled by automatic nervous system, which manages heart
muscles, smooth muscle and various glands in our body. [7]
These bodily reactions can be measured and monitored.
These signals are called as bio-signals. We can monitor the body
reactions from outside by use of Bio sensors and classify the bio
signals corresponding for each emotion.
Electrocardiogram (ECG). An electrocardiogram
(ECG) is a test which measures the electrical activity of the heart.
ECG is used to calculate the rate and regularity of heartbeats. Can
be recorded from above the chest or limbs. ECG measurements
from the limbs is less inconvenient but more vulnerable to
artifacts. [1]
We can measure heart rate (HR) and inter-beat intervals (IBI) and
thus we can determine the heart rate variability (HRV). A low
HRV can indicate a state of relaxation, whereas an increased HRV
can indicate frustration or state of mental stress.
Fig 2: (a) heart with Nervous system intervention (b) Heart rate
Variability HRV
Electromyography (EMG) is a technique used to record
electrical potential created by muscle membranes when there is an
electrical or neurological triggering. For example a high muscle
tension is created when there is stress or frustration. These signals
can be measured by placing the bio sensors over face or hands. [3]
Fig 3. Anatomy of hand (left), typical EMG signal (right)
Electrodermal response (EDR)-also called as skin conductivity
(SC) measures the conductivity of the skin. For example EDR
increases if the skin is sweaty. This signal is a sensitive indicator
of stress and for other stimuli. It is also used to classify between
conflicts-no conflict situations or classify between anger and fear.
The main disadvantage of this signal is that it is influenced by
external factors like outside temperature. Hence reference
measurements and calibrations are recommended. [3]

Fig 1: The Autonomic nervous system and the organs controlled
by it.
We study the following Bio-signals for Emotion recognition:
Figure 4: Anatomy of skin (left), typical EDR signal (right)
Skin temperature can be measured by determining the
temperature on the surface of the skin. For example under strain,
muscles are tensed, the blood vessels will be contracted and hence
the temperature will decrease. Similar to the Electro dermal

Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition

response the skin temperature also depends on external factors. adopted for many psychophysiological studies involving emotion
Furthermore it is a relatively slow indicator of changes in induction.
emotional state.
Blood volume pulse (BVP) is a measured by determining the
amount of blood currently running though the vessels, e.g. in the
finger of a test subject. A photoplethysmograph (PPG) is used to
measure BVP, it consists of a light source and photo sensor. It is
attached to the skin and the amount of reflected light, which
depends on the amount of blood, is measured. [1]
Respiration sensors measure how deep and fast a person is
breathing. This is measured by applying a rubber band around the
chest. Fast and deep breathing can indicate excitement such as
anger or fear but sometimes also joy. Rapid shallow breathing can
indicate tense anticipation including panic, fear or concentration.
Slow and deep breathing indicates a relaxed resting state while
slow and shallow breathing can indicate states of withdrawal, Fig.6 Illustration of example of emotion induction protocols'. Its"
passive like depression or calm happiness.
purpose is to induce status of 'sadness'[3]
We study the protocol utilizing a multimodal (audio, visual and
2. The Bio-Sensors cognitive) approach to evoke specific targeted emotional statuses,
and it was developed in collaboration with specialists from the
A Bio Sensor is an analytical device used to convert a
field of cognitive and physiological psychology (YANG et al.,
biological response into electrical signal.
2000). The emotion induction protocols are summarized in Table.
A preliminary test of the protocols was performed for 80 subjects
aged from seven to eight years. [3]

Figure 5. Main components of Biosensor

Figure 5. Examples of Bio-sensors.

Figure [5.1] shows the sensors used for measuring EMG. We
measure the muscle activity of the masseter muscle, [7] the
muscle movement is described to be reliable in that part of body.
Figure [5.2] shows a standard ECG sensor. In Figure [5.3] the
respiration sensor is shown applied to the chest while in Figure
[5.4] the skin conductivity, BVP and temperature sensor can be
seen applied to the fingers of the left hand (the non-dominant
hand should be used for measuring).[7]

3. Data Collection Experiments

Acquisition of emotion-specific physiological signal database:
The physiological signals were acquired using the MP100 system.
The sampling rate was fixed at 256 samples s 1 for all the
channels. Appropriate amplification and bandpass filtering were
performed. The subjects were requested to be as relaxed as
possible during this period. Subsequently, emotional stimulus was
applied and debriefing and recovery followed. [3]
Emotion induction protocol: We study that visual stimulation
using still images was not sufficient for effective emotion
induction, and we did not study the international affective picture
system (IAPS) developed by LANG et al. (1988), despite its being
4. Feature Extraction
4.1 Preprocessing, waveform detection and feature extraction:
The first necessary step was the detection of the characteristic
waveform and extraction of useful information-bearing features
for pattern classification. As shown in Fig. 7, the baseline values
of each component of the feature vectors were subtracted before
they were given to the classifier. Here, the baseline values mean
the components of feature vectors extracted from 50 s segments of
signals that were acquired without stimulus. [3]

Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition


Fig. 7 Overall structure of emotion recognition system [3] [4]
4.1.1. From ECG signals - RR interval and heart rate
variability:
Fig.8 Block diagram of information extraction module from ECG
Fig.8 above shows the block diagram of the feature extraction
module for the heart rate. Heart rate variability (HRV) contains
abundant information on the status of the autonomic nervous
system and can be derived from ECG or PPG. [3] Frequency-
domain features of HRV have also been considered to be
significant for the exploration of the autonomic nervous system in
many previous studies for cardiac function assessment and psycho
physiological investigation (DRUMMOND and QUAH, 2001;
MCCRATY et al., 1995).[3]
4.1.2 EMG Recordings and Data Reduction
Fig 9 supercilii and the zygomaticus muscles for EMG signal
acquisition. [5]
Facial EMG was measured by bipolar attached electrodes placed
over the corrugator supercilii and the zygomaticus major on the
left side of the face, according to the guidelines given by Fridlund
and Cacioppo (Figure 9). [5]
Submitted for Research Seminar on Emotion Recognition on 15.02.2012
Fig 10. Feature extraction from EMG signal [4]
4.1.3 Electrodermal activity:
Fig 11 (a) Typical waveform of EDA under emotional
stimulation.
(b) Output signal from detection module in (a).
(c) Block diagram of SCR detection module. Sampling rate of
traces in (a) and (b) is 21.3 samples s z.[3]
Electrodermal activity (EDA) was obtained by measurement of
the voltage between two electrodes across which a low-level
current was applied. Fig. 11 a shows a typical waveform of EDA
under emotional stimulation (after subtraction of mean EDA
level). Important features of EDA include the DC level and the
distinctive short waveforms that are indicated by arrows in Fig.
11. [3]
4.1.4 Skin temperature variation:
No special signal processing was necessary for the feature
extraction from the skin temperature (SKT). Although frequency-
domain analysis of the time-varying SKT has been reported
(SHUSTE~MAN and BARNEA, 1995), here the mean and
maximum values within 50 s intervals were used as the features of
SKT.[3]
5. Classification Experiments
After extracting the features as described in the previous section,
statistical classifier is used for learning the corresponding emotion
for a set of features with which it is presented. There are different
options for building such a classifier. Hilbert Haung transform,
wavelet transform, neural network classifier are some of the
widely used Classification methods.
5.1 Hilbert Haung transform
Huang et al. have proposed a new tool for the analysis of
nonlinear and non-stationary data called as the Hilbert-Huang
transform method (HHT)[2]. Fission process- Empirical Mode
Decomposition (EMD) decomposes a signal into basis functions
which are finite called the intrinsic mode functions. Fussion
process- After this fission process, the intrinsic mode functions
(IMFs) of interest are combined in an ad hoc or automated
fashion in order to provide a greater knowledge. [2].
http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition

Fig 12. HHT Fission and Fusion methods
Fission process- Each signal is decomposed into a finite
set of AM-FM mono components by using the Empirical Mode
Decomposition (EMD) which is the important part of the HHT.
Fusion process- For the next classification stage, the information
components of interest are then combined to create feature vectors
[2].
6. Results
Table 2. SVM Classification rates of four emotions for four bio
signals. (TD : Time Domain ; FD : Frequency Domain)[2]
The features from all methods ( the state-of-the-art method, the
fission process and the fusion process based HHT features
method) for four bio signals are combines and classified using
Support Vector Machine (SVM). The classification results are as
listed below
Table 3. Classification results [2].
7. Conclusion
In this paper we have surveyed research works using
Physiological sensors to recognize human emotions. In this
article, we discus about characteristic waveform detection, feature
extraction and pattern classification stages. The HHT Hilbert-
Huang transform method classified signals by 71 % for Baseline
features and 76% for fission method and 62% for fusion method.
[2]. Future Research scope: There are clearly more steps to take in
this research area. Research has to be done to improve the
accuracy of the network. This could be done either by adding
some meaningful features (the full feature set) that are computed
from the existing sensor signals or by using different bio-signals
such as EEG data. Also research has to be done to for
incorporation of different sources for emotion recognition such as
Submitted for Research Seminar on Emotion Recognition on 15.02.2012
video analysis, motion detection or emotion recognition from
speech signals to bring a real emotional dialog system to work.
References:
1. Rangaraj M. Rangayyan; Biomedical Signal Analysis – A
Case-Study Approach; IEEE Press 2002
2. C. Zong and M. Chetouani, "Hilbert-Huang transform based
physiological signals analysis for emotion recognition", in proc.
of IEEE International Symposium on Signal Processing and
Information Technology, pp. 334-339, 2009.
3. K. H. Kim I S.W. Bang 2 S.R. Kim 2Emotion recognition
system using short-term monitoring of physiological signals, 1
Department of Biomedical Engineering, College of Health
Science, Yonsei University, South Korea. 2 Human-computer
Interaction Laboratory, Samsung Advanced Institute of
Technology, South Korea
4. Cheng-Ning Huang Chun-Han Chen Hung-Yuan Chung, The
Review of Applications and Measurements in Facial
Electromyography,Department of Electrical Engineering, National
Central University, Chung-Li, Tiwan, 320 ROC
5. Anton van Boxtel,Facial EMG as a Tool for Inferring Affective
States, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The
Netherlands, a.vanboxtel@uvt.nl, Tilburg University, Tilburg,
The Netherlands,
6. Marco Tamietto , Emotional Contagion for Unseen Bodily
Expressions: Evidence from Facial EMG, Cognitive and Affective
Neurosciences Laboratory, Department of Psychology, University
of Torino, Torino, Italy, M.Tamietto@uvt.nl
7. Andreas Haag, Silke Goronzy, Peter Schaich, “Emotion
Recognition Using Bio-Sensors: First Steps Towards an
Automatic System” Jason Williams,Sony Corporate Laboratories
Europe, Hedelfinger Str. 61,D-70327 Stuttgart, Germany {Haag,
Goronzy, Schaich, Williams@Sony.de}
8. FATMA NASOZ, KAYE ALVAREZ, “Emotion Recognition
from Physiological Signals for Presence Technologies”, In
International Journal of Cognition, Technology, and Work -
Special Issue on Presence, Vol.6(1), 2003.
9.Lang, P.J.,Bradley, M.M., & Cuthbert, B.N. (2001).
International affective picture system (IAPS):Instruction Manual
and Affective Ratings. Technical Report A-5. Gainesville, FL.
The Research in Psychophysiology, University of Florida.
10. Center for the Study of Emotion and Attention [CSEA-NIMH]
(2001). The international affective picture system: Digitized
photographs. Gainesville, FL: The Center for Research in
Psychphysiology, Univerity of Florida
11. Barreto A., Heimer M., and Garcia M., “Characterization of
Photoplehtysmograpic Blood Volume Pulse Waveforms for
Exercise Evalution,” Proceedings 14th Southern Biomedical
Engineering Conference, Shreveport, Louisiana, April, 1995, pp.
220-223
12. Chrisite, “Multivariate Discrimination of Emotion-specific
Autonomic Nervous System Activity”, Master Thesis in Science
of Psychology, Virginia
13. Healy J. and Picard R., “Digital processing of Affective
Signals”, ICASSP 98
http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition