quality of multimedia experience past, present and future

1

ACM MultimediaOctober 22nd, 2009, Beijing, China

Quality of Multimedia Experience

Past, Present and Future

Prof. Dr. Touradj Ebrahimi

[email protected]

2


What is “quality” of multimedia content? How is multimedia content “quality” measured

today? What are trends in assessment of “quality” in

multimedia? What are the challenges ahead?

Today we will talk about…

3


Quality – a simple yet difficult concept

Like many human sensations quality is easy to understand but difficult to define

According to Wikipedia:– A quality (from Latin - qualitas) is an attribute or a

property. – Some philosophers assert that a quality cannot be

defined. – In contemporary philosophy, the idea of qualities and

especially how to distinguish certain kinds of qualities from one another remains controversial.

4


A fundamental yet largely under-investigated concept

Aristotle classified every object of human apprehension into 10 Categories– Substance, Quantity,

Quality, Relation, Place, Time, Position, State, Action, Affection

Aristotle 384 BC – 322 BC

5


Some definitions according to dictionary

Definition 1– General : Measure of excellence or state of

being free from defects, deficiencies, and significant variations.

– ISO 8402-1986 standard defines quality as "the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs"

6



Definition 2– Manufacturing : Strict and consistent

adherence to measurable and verifiable standards to achieve uniformity of output that satisfies specific customer or user requirements.

7



Definition 3– Objective : Measurable and verifiable aspect

of a thing or phenomenon, expressed in numbers or quantities, such as lightness or heaviness, thickness or thinness, softness or hardness.

8



Definition 4– Subjective : Attribute, characteristic, or

property of a thing or phenomenon that can be observed and interpreted, and may be approximated (quantified) but cannot be measured, such as beauty, feel, flavor, taste.

9


Definition according to ISO 9000

ISO 9000: a family of standards for quality management systems

Quality of something can be determined by comparing a set of inherent characteristics with a set of requirements– High quality: if characteristics meet requirements– Low quality: if characteristics do not meet all

requirements

Quality is a relative concept– Degree of quality

10


Quality – is in fact an elephant

The blind men and the elephant: Poem by John Godfrey Saxe

11


Quality of Service in computer networks and communications

Quality of Service (QoS) refers to a collection of networking technologies and measurement tools that allow for the network to guarantee delivering predictable results

12


Quality in QoS framework

Network QualityCapacity

CoverageHandoff

Link QualityBitrate

Frame/Bit/Packet lossDelay

User QualitySpeech fidelity

Audio fidelityImage fidelityVideo fidelity

13


Quality of Service in computer networks and communications

Quality of Service (QoS)– Resource reservation control mechanisms– Ability to provide different priority to different

applications, users, or data flows– Guarantee a certain level of performance

(quality) to a data flow

(Service) Provide-centric concept Tightly related to the concept of Mean

Opinion Score (MOS)

14


What is Mean Opinion Score (MOS)?

Widely used in many fields:– Politics/Elections– Marketing/Advertisement– Food industry– Multimedia– …

The likely level of satisfaction of a service or product as appreciated by an average user

Should be performed such that it generates reliable and reproducible results– Subjective evaluation methodology– More complex and difficult that it a priori seems

15


What is behind a MOS?

16


A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices:

Subjective evaluation

• Test/lab environment• Test material• Test methodology• Analysis of the data

17


Test/lab environment

Type of Monitors/Speakers and other test equipments Lighting /Acoustic conditions Laboratory architecture, background, … Viewing distance /Hearing position …

18


Test material

Meaningful content for the envisaged scenario/application– Typical content– Worst case content– …

p01 p06 p10 bike cafe woman

19


Test methodology

Subjects– Naïve or Expert?

Instructions– Which questions to ask subjects and how– Training

Presentation– Single or double stimulus– Sequential or simultaneous

Grading scale– Numerical– Categorical

…

20


• Test conditions and methodologies are specified in:

Recommendation ITU-R BT. 500-11 “Methodology for the subjective assessment of the quality of television pictures” (1974-2002).

Recommendation ITU-T P. 910 “Subjective video quality assessment methods for multimedia applications” (1999).

Recommendation ITU-R BT. 1788 “Methodology for the subjective assessment of video quality in multimedia applications” (2007).

Based on television scenario!

ITU Recommendations for test methodologies

21


Single Stimulus (SS)

Test methodology (I)

Non-categorical adjectival or numerical grading scale

5 Excellent 4 Good

3 Fair

2 Poor

1 Bad

5 Imperceptible

4 Perceptible but not annoying

3 Slightly annoying

2 Annoying

1 Very annoying

100

0

Excellent

Bad

Categorical adjectival grading scale: Categorical numerical grading scale:

“Rate from 1 to 11”

22


Double Stimulus Impairment Scale (DSIS)

Test methodology (II)

Categorical impairment grading scale:

5 Imperceptible

4 Perceptible but not annoying

3 Slightly annoying

2 Annoying

1 Very annoying

23


Double Stimulus Continuous Quality Scale (DSCQS)

Test methodology (III)

Sample 1Sample 2

Non-categorical adjectival or numerical grading scale:

100

0

Excellent

Bad

Sample 1 Sample 2

100

0

Excellent

Bad

24


Stimulus Comparison (SC)

Test methodology (IV)

Categorical adjectival comparison scale:

“same or different” much worse

worse

slightly worse

the same

slightly better

better

much better

Non-categorical judgement:

Much worse

Much better

25


Single Stimulus Continuous Quality Evaluation (SSCQE)

Test methodology (V)

(Very annoying)

(Imperceptible)

26


Simultaneous Double Stimulus for Continuous Evaluation (SDSCE)

Test methodology (VI)

(Much better)

(Much worse)(Reference) (Test sequence)

27


Analysis of the data

• Scores distributions across subjects is assumed to be close to normal distribution

• Outlier detection and removal• Mean Opinion Scores (MOS) and 95% confidence intervals

N

mMOS

N

i ijj

1

NNtCI j

j

),2/1(

mij = score by subject i for test condition j.

N = number of subjects after outliers removal.

t(1-α/2,N) = t-value corresponding to a two-tailed t-Student distribution with N-1 degrees of freedom and a desired significance level α (α=0.05 in our case).

σj = standard deviation of the scores distribution across subjects for test condition j.

28


What is behind a MOS?

29


Relationship between estimated mean values

• Hypothesis test to find out whether the difference between two MOS values are statistically significant

Two-sided t-test:

• t statistic:

• Decision rule to reject H0:

BA MOSMOSH :0

BAa MOSMOSH :

NN

MOSMOSt

BA

BAobs 22

),2/1(),2/( NttORNtt obsobs

30


MOS hypothesis testJPEG 2000 4:2:0

JPEG 2000 4:4:4

JPEG

JPEG XR MS

JPEG XR PS

JPEG 2000 4:2:0

JPEG 2000 4:4:4

JPEG

JPEG XR MS

JPEG XR PS

0.25 bpp

0.50 bpp

0.75 bpp

1.00 bpp

1.25 bpp

1.50 bpp

6

5

4

3

2

1

0Number of times

H0 is rejected

31


• Subjective tests are time consuming, expensive, and difficult to design

• Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired• Full reference metrics• No reference metrics• Reduced reference metrics

Objective quality metrics

32


FR, RR and NR scenarios Full Reference approach:

Reduced Reference approach:

No Reference approach:

Input/Reference signal

Output/Processed signal

signalprocessing

FR METRIC



signalprocessing

NR METRIC



signalprocessing

Features extraction

RR METRIC

33


Full Reference scenario Metrics which look at the fidelity of the signal when compared to an

explicit reference:

processed signal=

perfect quality reference signal+

error signal

MOS predictors based on fidelity measures

34


Examples Mean Square Error (MSE) Peak Signal to Noise Ratio (PSNR) Maximum Pixel Deviation (Linf) Weighted PSNR Masked PSNR Structural SIMilarity (SSIM) Multiscale Structural Similarity (MSSIM) Visual Information Fidelity (VIF) etc…

MOS predictors based on fidelity metrics

35


Peak Signal to Noise Ratio

Widely used because of its simplicity and ease in formalizing optimization problems!

For image and video data (Y component), a correlation of circa 80% reported when compared to subjective MOS evaluation

MSE)12(

log10PSNR2B

10

M

1y

N

1x

2ba y)](x,Imy)(x,[Im

MN

1MSE

where:

M, N = image dimensions Ima , Imb = pictures to compare

B= bit depth

36


PSNR for color images/video (I)

Several alternatives to compute PSNR for color images/video:

WPSNR = w1PSNR1 + w2PSNR2 + w3PSNR3

)MSEwMSEwMSE(w

1)(210log

332211

2B

10

WPSNR_MSE

WPSNR_PIX

M

y

N

xbbbaaa

B

)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(ImwMN

)(log

1 1

2332211332211

2

10 112

10

37


PSNR for color images/video (II)

Which color space to use? RGB Y’CbCr other?

Which weights to use? w1=w2=w3=1/3 w1=0.8, w2=w3=0.1 other?

38


PSNR for color images/video (III)

on R component:

bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)

on G component: on B component:

39


on Y’ component: on Cb component: on Cr component:

bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)

PSNR for color images/video (IV)

40


Human Visual System (HVS) based metrics simulating properties of the early stages of the HVS Examples: PSNR-HVS-M, etc… simulating high level features of the HVS Examples: Osberger’s metric, etc…

Better correlation with human perception. High complexity.

MOS predictors based on fidelity metrics

Computationalmodel of thevisual system

Visibility Map

41


PSNR-HVS-M

H

2B

10 MSE)12(

log10MHVSPSNR

7-M

1i

7N-

1j

8

1m

28

1ncijH )n,m(T)n,m(XKMSE

where: M, N = image dimensions K = constant

= visible difference between DCT coefficients of the original and distorted based on a contrast masking

Tc = matrix of correcting factors based on standard visually optimized

JPEG quantization tables B= bit depth

ij)n,m(X

42


Metrics based on the hypothesis that the HVS is highly adapted for extraction of structural information from the content of a still image or video. degradation of still images or video = perceived structural information

variation Structural Similarity by Wang et al.

MOS predictors based on fidelity measures

43


Mean SSIM (MSSIM)

12

22

1

12121 C

C2)Im,(Iml

22

22

1

22121 C

C2)Im,(Imc

€

SSIM(Im1,Im2) = l(Im1,Im2)[ ]αc(Im1,Im2)[ ]

βs(Im1,Im2)[ ]

γ)0,0,0(

• Luminance comparison function: (C1=constant)

• Contrast comparison function: (C2=constant)

• Structure comparison function:

321

32,121 C

C)Im,(Ims

(C3=constant)

44


MSSIM vs PSNR

PSNR = 24.9 dBfor all the images

Mean SSIM (MSSIM)

MSSIM=0.9168 MSSIM=0.6949MSSIM=0.7052

45


At times one is interested in specific types of distortions that occur in multimedia systems

Examples Bluriness Blockiness Ringing/Mosquito noise Jerkiness etc…

Specific distortion metrics

46


Blur metric

A perceptual quality blur metric without a reference image. Example:

Gaussian blurred image JPEG2000 compressed image

47


NR blur metric

48


NR blur metric

140 145 150 155 160 165 170 175 1800

50

100

150

200

250

P2 P1 P2' P4 P3 P4'

Pixel position

Pixel value

49


Correlation subjective ratings / NR blur metrics

4 5 6 7 8 9 10 11 12 13 140

1

2

3

4

5

6

7

8

9

10

Objective blur measure

Su

bje

cti

ve

blu

r ra

tin

g

Gaussian blurred images

4 5 6 7 8 9 100

1

2

3

4

5

6

7

8

9

10

Objective blur measure

Su

bje

cti

ve

blu

r ra

tin

g

JPEG 2000 images

96% correlation 85% correlation

50


Multimedia communication – a definition

Multimedia is about sharing experience (real or imaginary) with others

In a way it all started with story telling and wall drawing around the fire in the caves of early men

Modern multimedia systems are evolved versions of the good old story telling and wall drawing, which hopefully offer increasingly richer experience

The degree of richness of the experience is measured by Quality of Experience (QoE)

51


Evolutions versus Revolutions in multimedia

Evolution: A given modality improves itself in terms of Quality of Experience:– B&W TV– Color TV– Stereo and CD quality audio TV– HDTV

Revolution: New modalities and expression are introduced bringing new dimensions in Quality of Experience– Photography– Cinema– Internet

52


Some of the major milestones in multimedia

Story telling and cave drawing Books and written press Photography Telegraph Telephone Radio and music recording Cinema Television and video recording Internet (including VOIP, IPTV, etc.) Mobile communication Social networking (Web 2.0) (What is next?)

53


Quality of Service vs Quality of Experience

Quality of Service: Value of the average user’s experience richness estimated by a service/product/content provider

Quality of Experience: Value (estimated or actually measured) of a specific user’s experience richness

Quality of Experience is the dual (and extended) view of QoS problem

– QoS=provider-centric– QoE=user-centric

54


Factors impacting Quality of Experience

Context Context

55


Quality of Experience in networked multimedia

56


Quality Wheel

57


Trends in QoE

Digital world has (re-)discovered the notion of quality– Lower quality content is less and less tolerated by

end-users– Digital technology can now rival and even surpass

the old analog systems performance while remaining cost effective

Increasing interest in QoE– Extending from device-centric and system-centric

quality optimization to end-to-end and especially user-centric optimization

58


Trends in QoE community building

Increased interest in workshops and conferences around the notion of quality assessment and metrics– QoMEX: International Workshop on Quality of Multimedia

Experience (http://www.qomex.org)– VPQM: International Workshop on Video Processing and

Quality Metrics for Consumer Electronics (http://www.vpqm.org)– IMQA: International Workshop on Image Media Quality and its

Applications (http://www.mi.tj.chiba-u.jp/imqa2008/)

QoE is one of the issues referred to in future funding programs by the EC– Workshop on “The Future of Networked Immersive Media”

(http://cordis.europa.eu/fp7/ict/netmedia/workshop/ws20090610_en.html)

59


Trends in standardization

Standardization efforts in quality assessment and metrics– Video Quality Experts Group (VQEG)– ITU-T SG 12 (Performance, QoS and QoE)– ITU-R WP6C (Prog. production and quality assessment)– JPEG (Advanced Image Coding - AIC)– MPEG (High performance Video Coding – HVC)– …

60


Challenges ahead

Some key issues in QoE need to be better addressed– Content-dependent quality assessment methods and metrics – Context-dependent quality assessment methods and metrics– Quality assessment methods and metrics beyond AV (haptics, …)– Multi-modal quality assessment methods and metrics (AV, …)– 3D quality assessment methods and metrics (3D sound, 3D video, …)– HDR content quality assessment methods and metrics – Interaction quality metrics (closely related to usability)– Presence/immersion quality metrics– …

Need for Quality Certification Mechanisms of multimedia services and products– Similar in idea to ISO 9000 series

61


What does this all mean to you?

Era of user-centric multimedia has already started– It is not anymore sufficient to merely add new

features and functionalities to multimedia systems

– True added value in terms of impact on user’s experience of such features and functions should be demonstrated

– Quality of Experience plays a central role in this new game

62


Thank you for your attentionQuestions?

63


Acknowledgements

Some of the concepts and illustrations presented in this talk are the results of collaborations with individuals with whom I have had the pleasure of working. In particular:

– Francesca de Simone (EPFL/MMSPG)– Frederic Dufaux (EPFL/MMSPG)– Lutz Goldmann (EPFL/MMSPG)– Jong-Seok Lee (EPFL/MMSPG)– Andrew Perkis (NTNU/Q2S)– Ulrich Hoffmann (NTNU/Q2S)– Fitri Rahayu (NTNU/Q2S)

Part of the work presented here are the fruits of the research projects:

– EC funded NoE Petamedia– Swiss NSF funded NCCR IM2

quality of multimedia experience past, present and future

Documents