Evaluation of Noise Reduction in

Modern Mobile Devices

Valrie GAUTIER-TURBIN

Hans Wilhelm GIERLICH

Orange Labs
Products and Services

HEAD acoustics GmbH

(OLPS)
Communication and Services direction (COMSERV)
Solutions for Voice and Quality department (SVQ)

ETSI2012.Allrightsreserved

Summary

Chapter1
h
Chapter2
Chapter3
Chapter4
p

ETSI2012.Allrightsreserved

Subjectiveassessmentofnoisereduction
b
f
d
Objectiveevaluationofnoisereduction
Fundamentalsetuptocollectrecordings
Analysisofmeasurementofmobilephones
y
p

TheBasis:ITU
TheBasis:ITUTRec.P.835
TRec.P.835(11/2003)
Subjectivetestmethodologyforevaluatingspeechcommunicationsystemsthatincludenoisesuppressionalgorithm

Subjectiveevaluationofnoisereducedsignals:abidimensional
issue
Someusersfocusonbackgroundnoise(amountanddistortion)
Someusersfocusonthespeechsignal(distortion)

PrincipleofP.835:todrawlisternersattentiononbothtypesof
distortionbeforeprovidinganoverallqualityscore
Listenerssuccessivelyratetheprocessedsignalon
thespeechsignaldistortion,
thebackgroundnoiseintrusiveness
g
theoverallquality
Sub-sample 1
Signal or background noise rating

Sub-sample 2
Background noise or signal rating
Vote 2
BACKGROUND or SIGNAL

Vote 1
SIGNAL or BACKGROUND

Attending ONLY to the SPEECH SIGNAL, select the category

which best describes the sample you just heard.
The
5
4
3
2
1

ETSI2012.Allrightsreserved

SPEECH SIGNAL in this sample was

NOT DISTORTED
SLIGHTLY DISTORTED
SOMEWHAT DISTORTED
FAIRLY DISTORTED
VERY DISTORTED

Sub-sample 3
Overall quality rating

Attending ONLY to the BACKGROUND, select the category

which best describes the sample you just heard.
The
5
4
3
2
1

BACKGROUND in this sample was

NOT NOTICEABLE
SLIGHTLY NOTICEABLE
NOTICEABLE BUT NOT INTRUSIVE
SOMEWHAT INTRUSIVE
VERY INTRUSIVE

Vote 3
OVERALL QUALITY

Select the category which best describes the sample you

just heard for purposes of everyday speech communication.
The
5
4
3
2
1

OVERALL SPEECH SAMPLE was

EXCELLENT
GOOD
FAIR
POOR
BAD

Chapter2
Chapter
2

Objectiveevaluationofnoise
reduction

ETSI2012.Allrightsreserved

ETSIEG202396
ETSIEG
2023963
S
SpeechqualityperformanceinthepresenceofbackgroundnoisePart3:Backgroundnoisetransmission
h
lit
f
i th
fb k
d i P t3 B k
d i t
i i
objectivetestmethods
bj ti t t
th d

ObjectivetestmethoddevelopedtocalculateobjectiveMOS
forspeech,noiseandtheoverallqualityofatransmitted
for speech, noise and the overall quality of a transmitted
signalcontainingspeechandbackgroundnoise

similarlytoP.835:estimatedSMOS,NMOSandGMOS
basedonanaurallyadequateanalysis
useofP.501frenchsentencesforWB
use of P 501 English sentences for NB
useofP.501EnglishsentencesforNB
needs3inputsignals:clean,noisy(clean+backgroundnoise),processed

Generallyapplicabletonarrowbandandwidebandterminals
G
ll
li bl t
b d d id b d t
i l
Tobeusedinconjunctionwiththebackgroundnoisesdefined
d
id d i EG 202 396 1
andprovidedinEG2023961

ETSI2012.Allrightsreserved

ETSITS103106(2012
(201208)
S
Speechqualityperformanceinthepresenceofbackgroundnoise:Backgroundnoisetransmissionformobileterminals
h
lit
f
i th
fb k
d i B k
d i t
i i f
bil t
i l objectivetestmethods
bj ti t t
th d

Basedonnewsubjectivetestsconductedwithmodernmobile
terminalsin3GPP
terminals in 3GPP
DifferentreferencesystemcomparedtoEG2023963
Onlymobileterminalsused
useofnewspeechmaterial

ModificationofEG2023963:
NeuralnetworkinsteadofregressionforSMOScalculation
N
l
ki
d f
i f S MOS l l i
NewmappingforS, N andGMOSbasedonnewsubjectivetests

Differences to EG 2023963:
DifferencestoEG2023963:
restrictedscopecomparedtoEG2023963
applicableforNBandWB(AMRandAMRWBcoding,CDMAtestsongoing)
resultscomputedonapersentencebasisandaveragedoverallsamples

ETSI2012.Allrightsreserved

Chapter3

Fundamentalset
Fundamentalsetuptocollect
recordings

ETSI2012.Allrightsreserved

SetupbasedonETSIES
ETSIES202396
2023961
Objectivemodelsneed3inputs:clean,noisyandprocessed
clean:atourdisposal,signaltransmittedtotheHATSsmouth
noisy:mixtureofcleanandbackgroundnoise
useofdedicatedloudspeakerarrangementstatedinES2023961toreproduceinalab
environmentrealisticbackgroundnoiseconditions
noisysignalpickedupbyareferencemicrophonepositionedclosetotheprimary
microphoneofthemobileterminal

processed:mobileoutputofthenoisysignaltransmittedbythemobilephone
processed: mobile output of the noisy signal transmitted by the mobile phone
requiresanetworksimulator(controllednetwork)
Loudspeaker
rear right
rearright

Networksimulator
Loudspeaker
rearleft

Loudspeaker
frontleft

Reference
microphone
Subwoofer

Backgroundnoise
simulator

ETSI2012.Allrightsreserved

Loudspeaker
frontright

UpdateofEG202396
UpdateofEG
2023963
3
ES
ES202396
2023963
Fundamentalcalibrationandequalizationproceduresofthe
d
l lb
d
l
d
f h
backgroundnoisesimulationsystem
impactoftheloudspeakerspositioning:cornersshallbeavoided
impactoftheroom:noinfluenceonlyandonlyifproperequalization
useofstaticdelaysinsteadofautomaticdelays
nosharpfiltering
startequalizationfrom50Hztocoverwiththepresenceofwidebandterminals
start equalization from 50 Hz to cover with the presence of wideband terminals

interlabsrepeatablemeasuresonlyifpositioning,calibrationandequalizationstepsare
doneproperlyontheentirefrequencyband

Differenttypesofrealbackgroundnoiseavailable
insidecarnoise,insidebusnoise,insidetrainnoise,
inside car noise, inside bus noise, inside train noise,
publicplacesnoise,
workplacenoise,
8noisesselectedby3GPPandSA4forNoiseReductionevaluation
pub,road,crossroads,train,fullsizecar,cafeteria,mensa,office
9

ETSI2012.Allrightsreserved

Chapter4

Analysisofmeasurementsonmodern
mobilephones:selectedexamples

10

ETSI2012.Allrightsreserved

DescriptionofOrangedata(Widebandonly)

Noisereduction:useof6differentWideBandmobileterminals

from6differentmobileproviders(D1,D2,D6)
variousacousticdesigns
someequippedwithmonomicrophoneandotherswithdualmicrophones

Recordings in an acoustically treated room

Recordingsinanacousticallytreatedroom

setupinaccordancewithES2023961
useoftheACQUAandtheHAEBGNsystemsfromHEADacoustics
network simulator CMU200
networksimulatorCMU
200fromRohde&Schwarz
from Rohde&Schwarz

speechsamples:useof6talkers(3males,3females),2sentencespertalker

backgroundnoiseconditionslimitedtothe8noisesselectedby3GPPandGSMA

P.835subjectivetests

performedbyandatOrangeLabswithnaivelisteners(24votespersample)

288 votes per condition (6 talkers x 2 sentences/talker x 24 votes/sentence) smallconfidenceintervals

small confidence intervals
288votespercondition(6talkersx2sentences/talkerx24votes/sentence)

twoP.835tests:

onewithroad,train,fullsizecar,cafeteriaandofficenoises providedfor3QUESTretraining
onewithpub,mensaandcrossroadsnoises providedforvalidationoftheretrained3QUEST

11

ETSI2012.Allrightsreserved

Comparison ofobjectiveandsubjectiveresults
ofobjectiveandsubjectiveresults
OrangeWBdata
Orange WB data
SMOS
5
4,5

MOSscore

4
3,5

EG2023963

P.835

2,5

TS103106

2
1,5
D2

D3

D4

D5

D6

NMOS
5
4,5

Improvedrobustnesstonoise
Improved robustness to noise

MOSscore

4
3,5

EG2023963

P.835

2,5

TS103106

2
1,5
D1

D2

D3

D4

D5

D6

GMOS
4,5
4

MOSscore

35
3,5
EG2023963

P.835

2,5

TS103106

2
1,5
1
D1

ETSI2012.Allrightsreserved

D2

D3

D4

D5

D6

reductionsystems

withEG2023963,D1
and 6 were badly
andD6werebadly
estimated(upto1.0MOS
difference)

12

Averagedover8noise
conditions,goodpredictions
conditions,
good predictions
forall3dimensions(S,N,and
GMOS)withTS103106

1
D1

TS103106exhibitsmuch
betterperformancethan
EG2023963

Comparisonofobjectiveresults
Comparisonofobjectiveresults
Orange WB data
OrangeWBdata
A l i f bj i d f
Analysisofobjectivedatafrom52WBmobileterminals
52 WB
bil
i l
EG2023963ismorepessimisticonSMOS:TS103106,closertohumanperception,smoothes
performance (3.5 SSMOS
performance(3.5
MOS
4.1)

EG2023963providesNMOShigherthanTS103106(0.5MOS)
SMOS
4,5
4
3,5
3

EG2023963

2,5

TS103106

2
1,5
1
A B

G H

L M N O P Q R

T U V W X

Z A1 A2 C1 D1 E1 F1 G1 H1 I1 J1 K1 L1 M1 N1 O1 P1 Q1 R1 S1 T1 U1 V1 W1 X1 Y1 Z1

NMOS
OS
5
4,5
4
3,5
EG2023963

TS103106

2,5
2
1,5
1
A B

13

ETSI2012.Allrightsreserved

G H

L M N O P Q R

T U V W X

Z A1 A2 C1 D1 E1 F1 G1 H1 I1 J1 K1 L1 M1 N1 O1 P1 Q1 R1 S1 T1 U1 V1 W1 X1 Y1 Z1

Comparisonofobjectiveresults
Comparisonofobjectiveresults
HEAD acoustics WB data
HEADacousticsWBdata
A l i of16WBhandheld
Analysis
f 16 WB h d h ld HFTmobileterminals:Cafe
HFT
bil t
i l C f Noise63dB(A)
N i 63dB (A)
Wide rangeofSMOSwith EG2023963,typically lower than TS103106based tests
Wide rangeofNMOSwith
range of N MOS with EG2023963,typically
EG 202 396 3 typically lower than TS103106based
TS 103 106 based tests

5.0
G-MOS Range
G-MOS / Caf
N-MOS / Caf
S-MOS / Caf
S-MOS min
N-MOS min
S-MOS avg
N-MOS avg
GMOS avg

4.5
4.0
3.5
3.0
2.5
2.0
1.5

-D
ev
1
1
-D
e
M
v2
2
-D
ev
M
1
2
-D
e
M
v2
2
-D
ev
M
3
2
-D
ev
M
4
2
-D
e
M
v5
3
-D
ev
M
1
3
-D
ev
M
2
3
-D
e
M
v3
4
-D
ev
M
1
4
-D
ev
M
2
5
-D
e
M
v1
5
-D
ev
M
2
6
-D
e
M
v1
6
-D
ev
2

1.0

EG 202 306-3

14

ETSI2012.Allrightsreserved

---

---

TS 103 106

Comparisonofobjectiveresults
Comparisonofobjectiveresults
HEAD acoustics WB data
HEADacousticsWBdata
A l i f 16 WB h d h ld HFT
Analysisof16WBhandheldHFTmobileterminals:CarNoise69dB(A)
bil t
i l C N i 69 dB(A)
WiderrangeofSMOSwithEG2023963,typicallylowerthanTS103106basedtests
WiderrangeofNMOSwithEG2023963,typicallylowerthanTS103106basedtests
Wide rrange of N MOS with EG 202 396 3 typically lower than TS 103 106 based tests
HigherNMOSthanSMOSforsomedeviceswithEG2023963,SMOSalwayshigherwithTS103106

G-MOS Range
G-MOS / Car
N-MOS / Car
S-MOS / Car
S-MOS min
N-MOS min
S MOS avg
S-MOS
N-MOS avg
GMOS avg

5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5

-D
ev
1
1
-D
ev
M
2
2
-D
e
M
v1
2
-D
ev
M
2
2
-D
e
M
v3
2
-D
ev
M
4
2
-D
ev
M
5
3
-D
e
M
v1
3
-D
ev
M
2
3
-D
e
M
v3
4
-D
ev
M
1
4
-D
ev
M
2
5
-D
e
M
v1
5
-D
ev
M
2
6
-D
e
M
v1
6
-D
ev
2

1.0

EG 202 306-3

15

ETSI2012.Allrightsreserved

---

---

TS 103 106

Comparisonofobjectiveresults
Comparisonofobjectiveresults
HEAD acoustics WB data
HEADacousticsWBdata
A l i f 16 WB h d h ld HFT
Analysisof16WBhandheldHFTmobileterminals:RoadNoise74dB(A)
bil t
i l R d N i 74 dB(A)
Muchlowervaluesthanfortheothernoises
SimilarrangeofSMOSbutlowerforEG2023963thanforTS103106basedtests
Similar range of S MOS but lower for EG 202 396 3 than for TS 103 106 based tests
SimilarrangeofNMOSwithEG2023963,slightlylowerthanTS103106basedtests

G-MOS Range
G-MOS / Road
N-MOS / Road
S-MOS / Road
S-MOS min
N-MOS min
S MOS avg
S-MOS
N-MOS avg
GMOS avg

5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5

-D
ev
1
1
-D
ev
M
2
2
-D
e
M
v1
2
-D
ev
M
2
2
-D
e
M
v3
2
-D
ev
M
4
2
-D
ev
M
5
3
-D
e
M
v1
3
-D
ev
M
2
3
-D
e
M
v3
4
-D
ev
M
1
4
-D
e
M
v2
5
-D
ev
M
1
5
-D
ev
M
2
6
-D
e
M
v1
6
-D
ev
2

1.0

EG 202 306-3

16

ETSI2012.Allrightsreserved

---

---

TS 103 106

Conclusions

NewETSIstandardTS103106availableforevaluationof
mobilephonesinbackgroundnoise
mobile phones in background noise
RevisedETSIstandardES2023961availablewithbetter
definition of background noise simulation in wideband
definitionofbackgroundnoisesimulationinwideband
IngeneralSMOSestimationbasedonTS103106more
optimistic than with EG 202 396 3 less variation/less
optimisticthanwithEG2023963,lessvariation/less
discriminativebutbetteradaptedtoqualityperceptionof
mobile user
mobileuser
IngeneralNMOSestimationbasedonTS103106more
pessimistic than with EG 202 396 3 but very smilar in trend
pessimisticthanwithEG2023963butverysmilarintrend

17

ETSI2012.Allrightsreserved