1

Case study of PESQ performance in live wireless mobile VoIP environment

Zizhi Qiao (Motorola UK)Dr. Lingfen Sun (University of Plymouth, UK)Prof. Emmanuel Ifeachor (University of Plymouth, UK)

15/09/2008 PIMRC2008

2

Aim and Objectives

Aim is to investigate PESQ algorithm’s performance in wireless mobile VoIP environment

Objectives include:A live wireless mobile VoIP speech quality

test platform and its calibrationPESQ’s performance investigation, especially

time alignment related performance issues

3

We find out that in some cases, PESQ readings are significantly different from MOS scores. Here are a few examples:

A PESQ performance issue – why we do this research

Yes, PESQ is wrong in this case.

Time alignment error is the main issue.

Ref. Speech

PESQ=3.09

Deg. #23-38

PESQ=3.71

Deg. #25-46

But lets have a look of the test platform and its calibration before we go to PESQ’s accuracy issues, because the test platform forms

the basis of the PESQ performance investigation

PESQ scores 3.09 but MOS score is 3.6

Is PESQ wrong in this case? If so, Why?

4

A live wireless VoIP mobile test platform

5

Calibration of the live test platform

Volume settings (hardware / software) affect PESQ score

Calibrations are needed for each hardware or path settings

6

Case studies of PESQ’s accuracy issue

Two cases are discussed in details Silence gap caused wrong delay reading Time alignment caused wrong PESQ reading

They were all caused by time alignment method in PESQ

7

Silence gap caused wrong delay reading

•PESQ score on the top

•Relative delay on the bottom

•Wrong delay readings at the spike as pointed out

•A waveform analysis shows the PESQ delay reading is not correct

8

Silence gap caused wrong delay reading (cont.)

Degraded wave on the top

Wrong PESQ delay plot as shown on last slide

Silence gap caused wrong delay reading in PESQ

9

Time alignment caused wrong PESQ score

Positive shift in the waveform

Negative shift in the waveform

10

Time alignment caused wrong PESQ score (cont.)

PESQ’s reading for sample #23-38

And its wrong…

Next slide shows why PESQ was wrong

11

Time alignment caused wrong PESQ score (cont.)

We cut the sample and compared the chunks, each chunk has higher score than the original score, so we can conclude that PESQ was wrong and the error was caused by the delay shift.

12

Conclusions A live wireless mobile VoIP speech quality test platform

is established and calibrated.

A few PESQ performance cases has been discussed.

PESQ’s time alignment can not deal with some silence gaps and time shifting situations correctly.

More reliable time alignment method is necessary for PESQ to be used in new environment.

Before the new improved are standardised, more detailed investigation of PESQ results are necessary for each test, especially in new environment.

13

Thanks!

Q&A…