MOVING FROM PESQ TO POLQA

The Next-Generation Mobile Voice Quality Testing Standard

Dr. Irina Cotanis, on Moving from PESQ to POLQA

MOVING FROM PESQ TO POLQA

Why is a move from PESQ to POLQA recommended? (And, why will the move eventually be necessary?) What are the limitations of the POLQA technology? What are the main differences between POLQA and PESQ? What are the implications of moving to the new POLQA technology?

Dr. Irina Cotanis, on Moving from PESQ to POLQA

WHY MOVE FROM PESQ TO POLQA?

Reason 1. POLQA fixes known and recognized PESQ weaknesses. WB measurements in general Inaccuracy with CDMA codecs (e.g., EVRC) and, therefore, CDMA networks t k i in general l Sensitivity to certain GSM/WCDMA network conditions (DTX/Voice Activity ( y Detection in AMR codec) ) VoIP limitations (variable delay up to 1sec) Strong linear distortions (e.g., phones frequency shaping) as well as gain variations Limitations with speech processing (enhancement) devices (noise reduction reduction, automatic gain control)
Dr. Irina Cotanis, on Moving from PESQ to POLQA 3

WHY MOVE FROM PESQ TO POLQA?

Reason 2. POLQA has new operational features. Works in all three available bandwidths (NB, WB, SWB) and under different background noise conditions (street, car, restaurant, babble, impulse, etc.) W Works k with ith all ll new commercial i l and d standardized t d di d codecs d used di in diff different t technologies (GSM/WCDMA, LTE, VoIP, VoIP over IMS) such as AMR-WB, EVRC-WB, iLBC, AMB+, AAC, Skype / SLIK, G.711, G.729, etc. Allows measurements at various combinations of interfaces: electrical-electrical, acoustical-acoustical, electrical-acoustical, and acoustical electrical providing the opportunity to test terminals and acoustical-electrical hands-free applications

Dr. Irina Cotanis, on Moving from PESQ to POLQA

WHY MOVE FROM PESQ TO POLQA?

Reason 3. POLQA copes with new types of degradations. New codec conditions Single speech codecs and speech codecs working in tandem, packet loss and d concealment l t strategies t t i (PS), (PS) f frame errors and d bit errors ( (wireless), i l ) effects of coding on pre-noised speech p degradations g Network specific Interruptions (e.g., unconcealed packet loss or handover in GSM), front-end clipping (temporal clipping), amplitude clipping (overload, saturation), t ti ) time ti warping/time i /ti scaling li (VoIP (V IP over IMS)

Continued on next page

Dr. Irina Cotanis, on Moving from PESQ to POLQA 5

WHY MOVE FROM PESQ TO POLQA?

Reason 3. POLQA copes with new types of degradations. (continued) Voice Enhancement Devices (speech processing systems) implemented both in the CN as well as in the terminals Eff Effects t of f noise i reduction d ti systems t and d echo h cancellers ll on clean l speech h as well as on pre-noised speech Terminals Influence of time variant linear distortions (spectral shaping), non-linear distortions produced by microphone/transducer at acoustical interfaces Reverberations caused by hands-free test setups in defined acoustical environments

Dr. Irina Cotanis, on Moving from PESQ to POLQA

WHY MOVE FROM PESQ TO POLQA?

Reason 4. POLQA has been proven to over-perform PESQ. POLQA NB mode has shown increased performance of up to 25% as compared with PESQ NB Similar values haven been observed for POLQA SWB mode as compared with PESQ WB

Dr. Irina Cotanis, on Moving from PESQ to POLQA

WHAT ARE POLQAS LIMITATIONS?

STRICT LIMITATIONS. POLQA should be not used at all for:
Other dimensions of speech quality such as conversational aspects and talking quality Speech quality per call. POLQA is not intended to score longer sequences of speech. It is focused on prediction of quality for shorter speech utterances of 6 to 12 seconds Noisy listening environments. POLQA does not predict perceived speech quality in these environments; it is designed in accordance with P.800, ACR testing. Music (including multimedia) Evaluation of performance or ranking of voice enhancement devices (e.g., noise suppressors) Other technologies or components such as speech storage formats or non-telephony applications such as public safety networks or professional mobile radio connections
Continued on next page
Dr. Irina Cotanis, on Moving from PESQ to POLQA 8

WHAT ARE POLQAS LIMITATIONS? (continued)

POSSIBLE LIMITATIONS (scenarios on which POLQQ has not been tested/evaluated). In the following use cases, POLQA should be used carefully. Additional subjective testing might be needed for increased confidence. Other languages (e.g., (e g Arabic Arabic, or others not used for POLQA development) Longer speech samples Multi-dimensional audible speech p q quality y diagnosis g
NOTE. Work on Perceptual Approaches for Multi-Dimensional Analysis (PAMD) to predict perceptual dimensions of
g linked to the overall speech p q quality y in NB, WB, and SWB telecommunication scenarios is ongoing g g degradations within ITU. PAMD aims at providing more detailed information about individual quality dimensions as additional information to the POLQA overall MOS. It should be noted that this is different than diagnosing network problems based on speech quality parameters as provided by POLQA.

Dr. Irina Cotanis, on Moving from PESQ to POLQA

WHAT ARE THE MAIN DIFFERENCES BETWEEN POLQA & PESQ?

Reference speech material: 48kHz vs. 8kHz sampling frequency MOS-LQO output: POLQA score directly represents estimated MOS values on a subjective scale of 1 to 5 MOS, unlike PESQ, which used the P.862.1 mapping function for conversion to the MOS domain Dual operational modes: Narrowband NB (IRS filtering up to 3.4kHz; 8kHz sampling frequency of the reference sample) For backwards compatibility to PESQ recommended to be
run only for this scope

Super wideband SWB (Flat filtering up to 14kHz, 48kHz sampling frequency of the reference sample)
Continued on next page
Dr. Irina Cotanis, on Moving from PESQ to POLQA 10

WHAT ARE THE MAIN DIFFERENCES BETWEEN POLQA & PESQ? (continued)
Unique triple bandwidth output scale: SWB mode: NB, WB, and SWB tested samples presented on a single SWB scale (bandwidth being part of the transmission chains degradation) This has the advantage of direct comparison of different bandwidth scores.
NOTE: It is generally expected that the SWB unique scale (running a SWB reference sample) has the following transparency: NB: WB: Maximum MOS value 4.25 Maximum MOS value 4.5

SWB: Maximum MOS value 4.75

New extensively tested and validated languages: Chinese, Japanese, Czech Accuracy: POLQA proven to over over-perform perform PESQ (both NB and WB)
Dr. Irina Cotanis, on Moving from PESQ to POLQA 11

WHAT ARE THE IMPLICATIONS OF MOVING TO POLQA?

Possibility to achieve slightly lower spatial-temporal granularity of the speech quality measurements Longer processing time than PESQ due to algorithm complexity Larger L memory requirements i t d due t to 48kHz 48kH sampling li f frequency Proper understanding of the SWB unique scale, which provides three bandwidth scores on the same scale Backwards compatibility to PESQ Valid only for PESQ NB. Provides rankings rather than exact absolute values Comparison should be made only between PESQ NB and POLQA NB

Dr. Irina Cotanis, on Moving from PESQ to POLQA

12

CONCLUSIONS

Dr. Irina Cotanis, on Moving from PESQ to POLQA

13

CONCLUSIONS
The move from PESQ to POLQA is recommended (and will eventually be required) due to: POLQA fixes well-known weaknesses of PESQ POLQA copes with ith todays t d speech h codecs, d voice i enhancement h td devices, i and network conditions Q offers significantly g y higher g accuracy y POLQA POLQA technology has still some limitations that must be well understood and controlled

Continued on next page

Dr. Irina Cotanis, on Moving from PESQ to POLQA 14

CONCLUSIONS (continued)
Main differences between POLQA and PESQ: Reference speech sampling frequency Operational modes / bandwidths Unique triple bandwidth MOS LQO output scale Accuracy Implications of moving to POLQA: p y to PESQ Backwards compatibility Possibly lower spacetime granularity than PESQ Understanding the unique SWB output scale as well as its difference from the NB scale
Dr. Irina Cotanis, on Moving from PESQ to POLQA 15

THANK YOU!

Dr. Irina Cotanis, on Moving from PESQ to POLQA

16