International Telecommunication Union

ITU-T VICA Workshop22-23 July 2005, ITU Headquarters, Geneva

State of the Art ofState of the Art ofMultimedia Quality Multimedia Quality

Assessment MethodsAssessment Methods

Takanori HayashiNTT Service Integration Labs.

2ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Outline

o Present state of multimedia quality assessment methods

o Future trends of quality assessment research

• Multimodality• Multiparty• Wideband

o Examples of NTT’s studies on multimedia quality assessment for audiovisual communication services

3ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Relevant ITU-T/R Recommendationson Multimedia Quality Assessment

ObjectiveSubjective

J.148, P.931P.911, P.920Multimedia

J.144P.910, BT.500Video

P.563, P.862, BS.1387P.800, BS.562Audio

Quality assessment methodMedia

o Multimedia quality assessment is at a reasonably advanced stage.

4ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Existing Subjective Multimedia Quality Assessment Methods (P.911)

o Multimedia quality is assessed in a similar way to individual audio/video qualities.

• Absolute Category Rating (ACR)• Degradation Category Rating (DCR)• Pair Comparison Method (PC)• Single Stimulus Continuous Quality

Evaluation (SSCQE)o Assessment paying attention to the

cross-modal influences is important.• Interactions between differing quality

levels in different modalities

5ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-To Several category judgment scales are

used to evaluate multimedia quality.• Overall audiovisual quality• Individual audio/video qualities• Effort needed to interrupt• Communication difficulty• Acceptability of communication

o Communication quality depends on tasks used in conversational test.

o Assessment considering interactivity and usability is also important.

Existing Subjective Multimedia Quality Assessment Methods (P.920)

6ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Existing Concept Model of Objective Multimedia Quality Evaluation (J.148)

Auditory source

Visualsource

Differential delay

Visual quality

Auditory quality

Multimediaquality

integrationfunction

Task

Multimedia quality

Aq

Aq(Vq)

Vq(Aq)

Vq

o Basic components of an objective multimedia quality model are defined.

7ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Examples of Multimedia Quality Evaluation Model

o ITU-T P.911 Annex• MOSMM=C1MOSAMOSV+C2

o Other multimedia quality evaluation models have been proposed. [1-7]

• MOSMM=C1MOSA+C2MOSV+C3MOSAMOSV+C4! Constants vary with tasks (audiovisual

contents, service environments, etc.).

MOSMM: MOS for multimediaMOSA: MOS for audioMOSV: MOS for videoC: Constant

8ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Framework for ConductingQuality Assessment Research [8]

Sing

le m

ediu

mSi

ngle

med

ium

Media

WidebandWideband

Bandwidth

N:N(Multiparty)(Multiparty)

Communication style

InterdestinationInterdestination

synchronizationsynchronization

InterdestinationInterdestination

quality balancequality balance

Quality Quality assessment for assessment for

••multimodalitymultimodality

••multiparty multiparty

••wideband wideband

IntermediaIntermediasynchronizationsynchronization

IntermediaIntermediaquality balancequality balance

(VoIP)

1:11:1

Conventional research field

Future research field

Mul

tim

odal

Mul

tim

odal

9ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Key Words for Future Trends of Multimedia Quality Assessment (1/3)

o Multimodality

• Combination of multiple media such as audio, video, text, graphics, fax, and telephony in the communication of information

• Key points of quality assessment! Intermedia synchronization! Intermedia quality balance

10ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Key Words for Future Trends of Multimedia Quality Assessment (2/3)

o Multiparty

• Communication style extending from 1:1 to N:N (e.g., instant messaging, teleconferencing, and distributed collaboration services)

• Key points of quality assessment! Interdestination synchronization! Interdestination quality balance

11ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Key Words for Future Trends of Multimedia Quality Assessment (3/3)

o Wideband

• Telecommunications applications having more bandwidth available for higher-quality multimodal services

• Key points of quality assessment! Psychological factors: We need to assess

the richness of high-quality services not only on a one-dimensional scale, like MOS, but also on a multi-dimensional scale.

12ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Examples of NTT’s Studies on Quality Assessment for “Multimodality” (1/3)

o Interactive multimodal quality model considering conversational delay [9]

• MOSOA=(C1MOSMM+C2)(C3MOSR+C4)• MOSR=C5exp(-D/C6)+C7

! Constants C5, C6, and C7 depend on conversational task.

1

2

3

4

5

1 2 3 4 5Subjective MOSOA

Esti

mat

ed M

OS O

A

R2: 0.94MOSOA: MOS for overall qualityMOSMM: MOS for multimediaMOSR: MOS for responseD: DelayC: Constant

13ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Examples of NTT’s Studies on Quality Assessment for “Multimodality” (2/3)

o Influence of differential delay is different when DA>DV or DA<DV.[10]

Acceptable quality region for response

One-way delay for audio: DA [ms]

One

-way

del

ay f

or v

ideo

: D

V[m

s]

800

600

400

200

2000 400 600 800

Quality can be improved by synchronizing

DA: 100 msDV: 300 ms

DA: 300 msDV: 300 ms

Subjects: 4 expertsTask: Free conversation

14ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Examples of NTT’s Studies on Quality Assessment for “Multimodality” (3/3)

o Opinion model for audiovisual communication services is now being discussed in ITU-T SG12. [11]

Multimediaquality

Opi

nion

mod

el f

or

audi

ovis

ual c

om.

serv

icesVideo codec

Video size/resolutionVideo coding bit rateVideo packet loss rateVideo delayVideo jitter

Audio codecAudio echoAudio packet loss rateAudio delayAudio jitter

Video qualityparameters

Audio qualityparameters

15ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Examples of NTT’s Studies on Quality Assessment for “Multiparty” (1/2)

o Quality imbalance is one of the multiparty quality degradation factors.

Point 1

Point 2 Point 3

Three-point audiovisual communication

Other-point audiovisual quality:

MOS1-2

Other-point audiovisual quality:

MOS1-3

Overall audiovisual quality: MOS1

Good No GoodQuality

16ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Examples of NTT’s Studies on Quality Assessment for “Multiparty” (2/2)

o Overall quality is strongly affected by the inferior quality at another point. [12]

o Other-point audiovisual quality depends on conversation task or roles.

MOS1-2

MO

S 1-3

Overall audiovisual quality: MOS1

Higher

qua

lity

Higher quality

Hig

her

qual

ity

17ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Examples of NTT’s Studies on Quality Assessment for “Wideband” (1/3)

o Interactive multimodal quality can be evaluated using a multi-dimensional scale of psychological factors. [11]

Multimediaquality

Video qualityparameters

Audio qualityparameters

Perceptual quality indices

Parameter A

Parameter B

Parameter C

Psychological factor indices

Factor X

Factor Y

Factor Z

Video codecVideo size/resolutionVideo coding bit rateVideo packet loss rateVideo delayVideo jitter, …

Audio codecAudio echoAudio packet loss rateAudio delayAudio jitter, …

18ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Examples of NTT’s Studies on Quality Assessment for “Wideband” (2/3)

o Psychological factors were extracted by using the semantic differential (SD) technique and factor analysis. [13]

• Subject’s impression of an audiovisual communication service was evaluated on the basis of 25 pairs of bipolar adjectives on a seven-grade comparison scale.

Slow

NeutralSlightlyRatherExtremely ExtremelyRatherSlightly

Fast

Example of scale

19ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Examples of NTT’s Studies on Quality Assessment for “Wideband” (3/3)

o Multimedia quality was formulated as a function of two psychological factors expressing an aesthetic feeling and a feeling of activity. [13, 14]

Mul

tim

edia

qua

lity

g(f 1

,f2)

Video qualityparameters

Audio qualityparameters

Perceptual quality indices

Psychological factor indices

Blurring

Jerkiness

Distortion

Aesthetic feeling

f1(BR,PLRV,PLRA)Video coding bit rate: BRVideo frame rate: FRVideo packet loss rate: PLRVVideo delay: D

Audio packet loss rate: PLRAAudio delay: D

Delay

Lack of sound

Feeling of activity

f2(BR,PLRV, FR,D)

20ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

Conclusions

o Multimedia quality assessment is at an advanced stage.

o Perceptual quality assessment methodologies for multimedia communications systems of the next generation are being discussed.

o Three important characteristics of upcoming services have been revealed by recent studies on multimedia quality evaluation models:

• Multimodality, multiparty, and wideband.

21ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

References (1/2)[1] David S. Hands, “A Basic Multimedia Quality Model,” IEEE

Transactions on Multimedia, vol. 6, no. 6, pp. 806 - 816, Dec. 2004.

[2] C. Jones and D. J. Atkinson, “Development of opinion-based audiovisual quality models for desktop video-teleconferencing,”IEEE Sixth Int. Workshop on Quality of Service (IWQoS’98), pp. 196 - 203, May 1998.

[3] A. Joly, N. Montard, and M. Buttin, “Audio-visual quality and interactions between television audio and video,” Sixth International Symposium on Signal Processing and its Applications, Vol. 2, pp. 438 - 441, Aug. 2001.

[4] A. Watson, and M. A. Sasse, “Measuring perceived quality of speech and video in multimedia conferencing applications,”Proceedings of the sixth ACM international conference on Multimedia, Sept. 1998.

[5] ITU-T Contribution COM12-19-E, “Relations between audio, video, and audiovisual quality,” Dec. 1997.

[6] ITU-T Contribution COM12-61-E “Study of the influence of experimental context on the relationship between audio, video, and audiovisual subjective qualities,” Sept. 1998.

22ITU-T VICA Workshop

22-23 July 2005, ITU Headquarters, Geneva

ITU-T

References (2/2)[7] ITU-T Contribution COM12-64-E “Results of an audiovisual

desktop video teleconferencing subjective experiment,” Sept. 1998.

[8] A. Takahashi, H. Yoshino, and N. Kitawaki, “Perceptual QoSAssessment Technologies for VoIP,” IEEE Com. Magazine, pp. 28 - 34, July 2004.

[9] ITU-T Contribution COM12-D14-E, “Example of multimedia quality integration function for videophones,” Jan. 2005.

[10] S. Iai, T. Kurita, and N. Kitawaki, “Quality requirements for multimedia communication services and terminals – interaction of speech and video delays,” Globecom'93, pp. 394 – 398, 1993.

[11] ITU-T Contribution COM12-D13-E, “Proposal on basic concepts of a multimedia quality assessment model,” Jan. 2005.

[12] T. Kurita, “Effects of Conversation Roles on Quality of Multiparty Audiovisual Communication Services,” IEICE Tec. Rep. CQ, July 2005 (in Japanese).

[13] ITU-T Contribution COM12-D15-E, “Example of multimedia quality assessment model for videophones,” Jan. 2005.

[14] K. Yamagishi, and T. Hayashi, “Analysis of psychological factors for quality assessment of interactive multimodal service,”Electronic Imaging 2005, 5666-15, pp. 130 - 138, Jan. 2005.