drive testing. aqua. pvqa
TRANSCRIPT
Drive Testing. AQuA. PVQA.
Valeri Sitnikov
Sevana Oü
Tallinn, Estonia
+3726170331
End-to-End Drive Test
GSM,
WCDMA,V
oLTE,
VoWiFi,
RCS
Codec
(NB, WB)
Drive Tester
Voice Core, PackeCore, Evolved Packet Core
AQuA
PVQA
Drive Test Analysis
AQuA Metrics
MOS score according to P.800 – reliable objective MOS scores from 1 to 5
Percentage of sound quality similarity between reference and test audio
Duration distortion
Delay or advancing of audio signal activity
Sound signal activity mistiming
Signal spectrum damages
Signal spectrum vibration
Unidirectional signal spectrum analysis: amplification or attenuation
Most affected signal spectrum: low, medium, high bands
Average pitch
Pitch deviation
Pitch frequencies distribution distortion
Quantization steps distribution distortion
Amplitude clipping level
RMS
Sound signal: Minimum Energy, Maximum Energy, Average Energy
Sound signal: Min. Sample, Max. Sample, Average Sample
Drive Tester
Android ported NDK library, developed for Samsung phones
Linux, Windows compatible application/library, outperforming existing algorithms in performance
Any Linux-like operating system with standard C++ compiler
PESQ. POLQA. AQuA. Feature Comparison Table.
Feature PESQ POLQA AQuA
Measurements in wideband audio
NO YES YES
Accurate measurements in CDMA networks
NO YES YES
VAD detection in AMR codec
NO YES YES
GSM/WCDMA accurate measurements
NO YES YES
Variable delay up to 1 sec in VoIP
NO YES YES
PESQ. POLQA. AQuA. Feature Comparison Table.
Feature PESQ POLQA AQuA
Strong linear distortions NO YES YES
Limitations with speech enhancement devices (noise
reduction, gain control)
YES NO NO
Works in NB (up to 8 kHz Sample Rate) bandwidths
NO YES YES
Works in WB (up to 16 kHz Sample Rate) bandwidths
NO/YES YES YES
Works in SWB (above 16 kHz Sample Rate) bandwidths
NO YES YES
PESQ. POLQA. AQuA. Feature Comparison Table.
Feature PESQ POLQA AQuA
Works with audio sampling rate higher than 48kHz
NO NO YES
Works with all standard and non-standard codecs audio (AMR-WB, EVRC-WB, iLBC, AMB+, AAC, SLIK,
G.711, G.729)
NO YES YES
Works with all technologies (GSM, WCDMA, LTE, VoIP, VoIP
over IMS)
NO YES YES
Can be used in testing terminals and hand-free applications
YES YES YES
Considers codec degradations (tandem codecs, packet loss,
frame errors, bit errors)
NO YES YES
PESQ. POLQA. AQuA. Feature Comparison Table.
Feature PESQ POLQA AQuA
Customizable perceptual model and algorithm
NO NO YES
Considers network specific degradations
NO YES YES
Copes with P.800 MOS NO YES YES
Talking quality YES NO NO/YES
MOS score for long sequences of speech (more than 30
seconds)
NO NO YES
PESQ. POLQA. AQuA. Feature Comparison Table.
Feature PESQ POLQA AQuA
Noisy listening environments NO NO YES
Music clips NO NO YES
Noise suppressors NO NO YES
Non-telephony applications YES NO YES
Language independent YES NO YES
PESQ. POLQA. AQuA. Feature Comparison Table.
Feature PESQ POLQA AQuA
Multi-dimensional audio(Works with Stereo)
NO NO YES
Unified perceptual model for all types of audio
(speech, music)
NO NO YES
Able of working with long audio(1 hour and longer)
NO NO YES
AQuA has some key features vital for sound quality evaluation during deployment and operation of VoLTE, VoWi-Fi, RCS:
Language independentLong test calls support
Ability to work with Stereo sound
PVQA Metrics
MOS score according to P.800 – reliable objective MOS scores from 1 to 5
Impairments detection (reasons for QoE drop down):
Noise
DeadAir
Amplitude clipping
VAD Clipping
Clicking
Crackling
Fuzzy voice
Echo
DTMF
By just waveform analysis PVQA is able to discover such network issues as
Packet loss (DeadAir + Crackling)
Hardware issues (Amplitude clipping, VAD clipping)
PVQA Analysis
• Non-intrusively calculated MOS score with standard deviation of 0.6 MOS to intrusive AQuA test score
• CSV reports on impairments that caused sound quality degradation
Time; Average; SNR-00; DeadAir-01; DeadAir-00; AMPC-00; VADC-00; Click-00;
0.000 : 0.799; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
0.799 : 1.598; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
1.598 : 2.397; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
2.397 : 3.196; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
3.196 : 3.995; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
3.995 : 4.794; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
4.794 : 5.593; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
5.593 : 6.392; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
6.392 : 7.191; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
7.191 : 7.990; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
7.990 : 8.789; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
8.789 : 9.588; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
9.588 : 10.387; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
10.387 : 11.186; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
11.186 : 11.985; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
11.985 : 12.784; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
12.784 : 13.583; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
13.583 : 14.382; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
14.382 : 14.882; 0.013333; 0.000000; 0.080000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
Time; Average; SNR-00; DeadAir-01; DeadAir-00; AMPC-00; VADC-00; Click-00;
0.000 : 0.799; 0.014583; 0.000000; 0.087500; 0.000000; 0.000000; 0.000000; 0.000000; Ok
0.799 : 1.598; 0.018750; 0.000000; 0.112500; 0.000000; 0.000000; 0.000000; 0.000000; Poor
1.598 : 2.397; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
2.397 : 3.196; 0.085340; 0.000000; 0.425000; 0.000000; 0.000000; 0.050000; 0.037037; Poor
3.196 : 3.995; 0.039583; 0.000000; 0.225000; 0.000000; 0.000000; 0.012500; 0.000000; Poor
3.995 : 4.794; 0.018750; 0.000000; 0.100000; 0.000000; 0.000000; 0.012500; 0.000000; Poor
4.794 : 5.593; 0.034395; 0.000000; 0.162500; 0.000000; 0.000000; 0.025000; 0.018868; Poor
5.593 : 6.392; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
6.392 : 7.191; 0.055228; 0.000000; 0.300000; 0.000000; 0.000000; 0.012500; 0.018868; Poor
7.191 : 7.990; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
7.990 : 8.789; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
8.789 : 9.588; 0.043750; 0.000000; 0.225000; 0.000000; 0.000000; 0.037500; 0.000000; Poor
9.588 : 10.387; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
10.387 : 11.186; 0.114623; 0.000000; 0.612500; 0.000000; 0.000000; 0.037500; 0.037736; Poor
11.186 : 11.985; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
11.985 : 12.784; 0.014583; 0.000000; 0.075000; 0.000000; 0.000000; 0.012500; 0.000000; Ok
12.784 : 13.583; 0.004167; 0.000000; 0.025000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
13.583 : 14.382; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok
14.382 : 14.832; 0.014815; 0.000000; 0.066667; 0.000000; 0.000000; 0.022222; 0.000000; Ok
Time; Average; SNR-00; DeadAir-01; DeadAir-00; AMPC-00; VADC-00; Click-00;
0.000 : 0.799; 0.014583; 0.000000; 0.087500; 0.000000; 0.000000; 0.000000; 0.000000; Ok
0.799 : 1.598; 0.018750; 0.000000; 0.112500; 0.000000; 0.000000; 0.000000; 0.000000; Poor
1.598 : 2.397; 0.087500; 0.000000; 0.500000; 0.000000; 0.000000; 0.025000; 0.000000; Poor
2.397 : 3.196; 0.148920; 0.000000; 0.812500; 0.000000; 0.000000; 0.062500; 0.018519; Poor
3.196 : 3.995; 0.093750; 0.000000; 0.525000; 0.000000; 0.000000; 0.037500; 0.000000; Poor
3.995 : 4.794; 0.035417; 0.000000; 0.187500; 0.000000; 0.000000; 0.025000; 0.000000; Poor
4.794 : 5.593; 0.133333; 0.000000; 0.712500; 0.000000; 0.000000; 0.087500; 0.000000; Poor
5.593 : 6.392; 0.077083; 0.000000; 0.400000; 0.000000; 0.000000; 0.062500; 0.000000; Poor
6.392 : 7.191; 0.117728; 0.000000; 0.637500; 0.000000; 0.000000; 0.050000; 0.018868; Poor
7.191 : 7.990; 0.036900; 0.000000; 0.177215; 0.000000; 0.000000; 0.025316; 0.018868; Poor
7.990 : 8.789; 0.070833; 0.000000; 0.375000; 0.000000; 0.000000; 0.050000; 0.000000; Poor
8.789 : 9.588; 0.096836; 0.000000; 0.500000; 0.000000; 0.000000; 0.062500; 0.018519; Poor
9.588 : 10.387; 0.052083; 0.000000; 0.300000; 0.000000; 0.000000; 0.012500; 0.000000; Poor
10.387 : 11.186; 0.125000; 0.000000; 0.712500; 0.000000; 0.000000; 0.037500; 0.000000; Poor
11.186 : 11.985; 0.139506; 0.000000; 0.750000; 0.000000; 0.000000; 0.050000; 0.037037; Poor
11.985 : 12.784; 0.132311; 0.000000; 0.750000; 0.000000; 0.000000; 0.025000; 0.018868; Poor
12.784 : 13.583; 0.128145; 0.000000; 0.700000; 0.000000; 0.000000; 0.050000; 0.018868; Poor
13.583 : 14.382; 0.086478; 0.000000; 0.475000; 0.000000; 0.000000; 0.025000; 0.018868; Poor
14.382 : 14.832; 0.107407; 0.000000; 0.600000; 0.000000; 0.000000; 0.044444; 0.000000; Poor
PVQA Root Cause Analysis
VAD clipping related to shortening of voice snippets due to failures in voice activity detection, the so-called squelch
Amplitude clipping with noise - form of waveform distortion that occurs when an amplifier is overdriven and attempts to deliver an output voltage or current beyond
its maximum capability.
PVQA Pattern Discovery
There are two types of patterns PVQA allows to register and disover:
• Network specific impairments (prevailing of clipping, noise, or signals with
crackling)
• Combination of impairments specific for certain events, e.g. packet loss
Packet Loss pattern
Pitch
o AQuA offers reliable end-to-end sound quality assessment for both NB and WB audio providing reasonable MOS scores according to P.800.
o AQuA is working on laptops, servers and Android devices providing high performance in MOS calculation and analysis.
o AQuA can be ported to other systems that have standard C++ compiler (Linux-type operating systems, iOS)
o PVQA provides reliable approach for non-intrusive MOS score calculation
o PVQA provides additional information on impairments that cause voice quality degradation:
o Root Cause Analysis
o Impairments Pattern Discovery
o Packet loss detection by waveform analysis