ITU-T Study Group 12:Performance, Quality of Service and

Quality of Experience

Highlights of January 2017 meetingKwame BAAH-ACHEAMFUOR

Study Group 12 Chairman

Geneva, 10-19 January 2017

2

WP1/12: Terminals and multimedia subjective assessment

Highlights of January 2017 meeting

Lars Birger NielsenWP1/12 Chairman

Gunilla BerndtssonWP1/12 Vice-Chairman

4

P.381 “Technical requirements and test methods for the universal wired headset or headphone interface of digital mobile terminals”

Produced by Q.3 (Speech transmission characteristics of communication terminals for fixed circuit-switched, mobile and packet-switched (IP) networks)Requirements for electroacoustic test and corresponding test methods – incl. EN 50332-1 and 2Specification of pin assignments of a socket connector with three/four contact pointsFor characterisation of headset/headphones as well as the interface of digital wireless terminalsP.381: one microphoneRevision for support of super-wideband and fullband

Revised Recommendations Consented

5

P.11x0 “In vehicle Recommendations” Produced by Q.4 (Hands-free communication in vehicles)Revised Recommendation P.1100 (NB), P.1110 (WB) and P.1140 (eCall)

Measurement of Delay was aligned with 3GPP specificationsAdaptation of a new background noise equalization methodAlignment of descriptions and editorial updates

New Recommendation P.1120 (SWB + FB)

Recommendations Consented

6

P.501 “Test signals for use in telephonometry” Produced by Q.6 (Analysis methods using complex measurement signals including their application for speech enhancement techniques and hands-free telephony)Texts of amendments have been moved into the P.501 documentAll speech samples and other samples will become available for download once the approval process of P.501 has been concluded.

Revised Recommendations Consented

7

New Recommendations Consented

P.1310 “(Spatial audio meetings quality evaluation”

Produced by Q.10 (Conferencing and telemeeting assessment)Provide guidance on appropriate subjective testing methods for evaluating of spatial audio telemeeting systems – w/wo videoMethods for spatial audio on how to

Collect subjective quality ratingsAssess accuracy of the spatial alignment of audiovisual systemsObtain task performance indicators for the intelligibility of concurrent speakersMeasure communication effectiveness using task performanceObtain task performance indicators for the cognitive load experienced

Method Annex A Annex B Annex C Annex D Annex E

Rendering Scenario

Audio-only or audio component of an audiovisual system

Audiovisual Test Focus Overall System

Individual Components

Target Variables

Overall Quality (3 (3

Media Quality

Non-spatial quality

Spatial quality

Communication quality (2 (2 (2

Measurement Sensitivity

Sufficient (test added value of spatial audio)

High (differentiate variants of spatial audio)

1)

Test Paradigm

Listening-Only (Non-interactive) Test

Conversation Test

Measurement Paradigm

Subjective Ratings

Performance Measures

Notes:

1) Depending on the actual implementation of the method details, e.g. training phase, it is possible to achieve high measurement sensitivity. 2) These methods (indirectly) measure individual aspects of communication quality. 3) These methods are not measuring overall quality, but they may be combined with subjective ratings of overall quality.

8

New work items

Two new work items has been started

New work item on “Transmission characteristics for In Car Communication” or P.ICC

New work item on “Subjective methodology for Speech and Audio multi-channel Test” or P.SAT

9

Rapporteurs & SG12 MeetingsQ.4/12 “Objective methods for speech and audio evaluation in vehicles”

Date: August 2nd

Place: Geneva, SwitzerlandTopic: To progress on P.ICC

Q.5/12 “Telephonometric methodologies for handset and headset terminals”

Date: May 29th and 30th

Place: Bern, SwitzerlandTopic: To progress on P.TBN and P.Loudness

SG12 meetingDate: September 19-28 (2017)Place: Geneva, Switzerland

WP1/12: Terminals and multimedia subjective assessment

Highlights of January 2017 meeting

Lars Birger NielsenWP1/12 Chairman

Gunilla BerndtssonWP1/12 Vice-Chairman

END of WP1 PRESENTATION

WP2/12: Objective models and tools for multimedia quality

Highlights of January 2017 meetingVincent Barriac

WP2/12 Chairman

Ludovic MalfaitWP2/12 Vice-Chairman

List of Questions under study

Q9/12 Perceptual-based objective methods for voice, audio and visual quality measurements in telecommunication services

Q14/12 Development of parametric models and tools for multimedia quality assessment of packet-based video services

Q15/12 Parametric and E-model-based planning, prediction and monitoring of conversational speech quality

Q16/12 Framework for diagnostic functions

Q19/12 Objective and subjective methods for evaluating perceptual audiovisual quality in multimedia services

12

Q.9: summary of discussion (1/2)

13

P.863A modified version of P.863 that minimizes the under-prediction in case of extended micropauses was introduced. This version is available for testing to all interested parties in ITU-T and commercial POLQA customers.The results of a P.800 ACR subjective test focusing on the evaluation of EVS and Opus coding was introduced and discussed. Contributions providing additional experimental results would be greatly appreciated to verify the performance of P.863 and other objective measures on Opus and other proprietary codecs.

Q.9: summary of discussion (2/2)

14

P.AMD and P.SAMDNext SG12 meeting is targeted as the submission date for candidate models for both sets A (5 dimensions) and B (7 dimensions).For the new work item P.SAMD the existing Requirement Specification is extended to cover both P.AMD and P.SAMD in a single document.

P.SPELQThe latest results are looking promising for narrowband databases. Up to now, there are no published training results for super-wideband databases. A stable version of the P.SPELQ model is expected to be submitted to TSB before the next SG12 meeting so the evaluation can start

Q.9: outputs

15

No liaison statementOne document agreed:

Supplement for P series addressing P.863 and P.863.1: “Application of P.863 and P.863.1 for speech processed by blind bandwidth extension approaches”, available as TD163 rev 1.

One interim meeting is requestedProposed time frame is June / Beginning of JulyMain topics for discussion: P.AMD/P.SAMD and probably P.SPELQ.Exact date and location will be discussed and agreed via the Q9/12 reflector. Discussion topics will be updated according to the submitted contributions.

Q.14: summary of discussion (1/2)

16

P.1203: Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport

amendment with the final performance figures for all four modes, concluding the P.NATS phase 1 work

New work item for a new recommendation being the extension of the P.1201 Recommendations

This new work item focuses on UDP-based video for new codecs and higher video resolutions.

Q.14: summary of discussion (2/2)

17

New study item AVHD-AS/P.NATS phase 2 (joint project with VQEG)

extension of the scope of P.1203 with new video codecs and resolutions (including UHD), inclusion of a larger variety of model types (new: pixel-based)11 proponents have responded positively to the provisional call for participation.44 subjective test databases to be created

device types and viewing distance discussed and agreeddistribution of short vs. long sequence databases set to 70/30, training vs validation databases to 50/50 for short sequence databases and 25/75 for long sequence databases.

Q.14: outputs

18

One document agreedAmendment to P.1203 (available as TD139)

No liaison statementsTwo Rapporteur Meetings are requested

Q.13, 14 and 17/12 (“Q.44”) plus VQEG AVHD (to discuss AVHD / P.NATS Phase 2): End of March with the most likely dates 22-24 March 2017, hosted by Deutsche Telekom / T-LabsQ.14/12 / AVHD-AS meeting on AVHD/P.NATS Phase 2 at VQEG meeting to be held 8-12 May 2017 in Los Gatos, California, USA, hosted by Netflix

Q.15: summary of discussion

19

Wideband extension to G.107.1The decision has been taken to work on a basic SWB version of the E-model. Will be re-used by Q.13 for G.1070

P.CQOThe data received on the validation of the underlying subjective procedure (P.CQS) was considered very useful, but should be augmented by results from other test labs, involving different test set-ups, participant groups, and potentially languages to substantiate the subjective method.

Q.16: summary of discussion

20

P.TCA: Technical Cause AnalysisIt was proposed to draft a Recommendation that provides P.TCA degradation classes, their description and exemplary audio samples. This Recommendation would provide the industry with a reference for common terminology.

E.FINAD: Framework for Intelligent Network Analytics and Diagnostics

A new work item is proposed to recommend a framework for identifying anomalies and diagnosing root cause from data collected from production networks such as logs, statistics, alerts, etc…

Q.19: summary of discussion

21

J op-tr (Methods for Optimizing Bitrates and Transmission Resolution by Considering Display Characteristics and Available Bandwidth)

New progress based on experimental results and statistical analyses provided in C 67

showed that equivalent perceptual video quality could be provided with reduced bandwidth by considering the display characteristics (size and resolutions).

J.norefPotential interaction with Q.14 (P.NATS phase 2) in the near future

test data of P.NATS-Phase 2 could be re-used for the validation of J.norefpossibility to any model originating from Q.19 to be submitted as candidate models to P.NATS phase 2 competition and work itemdistinction between scopes: P.NATS phase 2 work item is focused on TCP based video, while J.noref work item originates from broadcast type of video and includes UDP based video and packet loss.

IRG AVQA

22

Decision to continueCo-chairman on behalf of SG12 unchanged: Jens Berger (Q.9 rapporteur)One liaison statement sent to SG9 (available as TD 160)

WP2/12: Objective models and tools for multimedia quality

Highlights of January 2017 meetingVincent Barriac

WP2/12 Chairman

Ludovic MalfaitWP2/12 Vice-Chairman

END OF WP2 PRESENTATION

WP3/12: Multimedia QoS and QoE

Highlights of January 2017 meetingPaul Coverdale

WP3/12 Chairman

Al MortonTiago Sousa Prado

WP3/12 Vice-Chairmen

25

New Recommendations Consented

E.847 (G.PoiCong) “QoS norms for TDM Interconnection between Telecom Networks”

This Recommendation analyses and identifies QoS parameters for TDM interconnection between telecom networks, which are needed to facilitate effective interconnections with reasonable traffic handling capacities.

26

New Recommendations Consented

Y.1545.1 (Y.FMIPQoS) “Framework for Monitoring the QoS of IP network services”

This Recommendation is a diagnostic reference for IP network quality of service monitoring, and primarily as a guide to assist regulators monitor the quality of service of internet that is provided by service providers (although subscribers and network service providers may also derive benefit).

27

New Recommendations Consented

E.811 (E.QMME) “Quality Measurement in Major Events”

This Recommendation addresses the quality assessment of mobile broadband and voice services provided during major events, such as FIFA World Cups and Olympic and Paralympic Games, by creating a useful international reference to be considered by operators and regulators when preparing to host such events.

28

Amended Recommendations Consented

E.802 “New Annex A on guidelines on selection of representative samples”

provides guidance on the selection of representative samples in the measurement of QoS parameters

29

Liaisons

To SG11 and TSAG regarding the work item Y.FMIPQoS (now Y.1545.1)

We have consented Y.1545.1The WTSA-16 guidance indicates that SG 12 has clear responsibility for various aspects of QoS, including measurement methodologies

30

Liaisons

To 3GPP on KPIs for mobile servicesrequests information and appropriate references related with Voice and Data QoS KPI thresholds on mobile networks.

31

Liaisons

To SG16, VQEG, MPEG, 3GPP SA4Informs about a new study item on quality of experience (QoE) for virtual reality (VR) services (G.QoE-VR)

32

Liaisons

To SG2 on study of website traffic characterization

Informs that we have removed G.MFWT from our work items

33

Liaisons

To SG5 on proposed collaboration on accuracy and availability requirements regarding soft error requirements

Recommends to recognize the evolution to VNF-based telecom architectures in K.soft_des

New work items

G.ViLTEEnd-to-end QoS for Video Telephony over 4G mobile networks

E.RQUALStrategies to Establish Quality Measurement Frameworks

E.NetPerfRankStatistical Framework for QoE Centric Benchmarking Scoring and Ranking

34

New work items

E.RQSTVoice and Data QoS KPI thresholds on mobile networks

G.QoE-VRQoE for Virtual Reality

35

36

Interim meetings

To be co-located with Q14/12Question 13/12

One meeting requestedTentatively March 2017 in Berlin

Question 17/12One meeting requestedTentatively March 2017 in Berlin

WP3/12: Multimedia QoS and QoE

Highlights of January 2017 meetingPaul Coverdale

WP3/12 Chairman

Al MortonTiago Sousa Prado

WP3/12 Vice-ChairmenEND OF WP3 PRESENTATION

Next Study Group 12 meeting

19-28 September 2017, Geneva, Switzerland

SG12 Website: www.itu.int/go/tsg12

38

39