on qoe metrics and qoe fairness for network & traffic management

Prof. Dr. Tobias Hoßfeld

Chair of Modeling of Adaptive Systems (MAS)Institute for Computer Science and Business Information Systems (ICB)University of Duisburg-Essen

www.mas.wiwi.uni-due.de

On QoE Metrics and QoE Fairness for Network & Traffic Management

Tobias HossfeldLea Skorin-KapovPoul Heegaard

05/01/2023 2

QoE Metrics and QoE Fairness

System

Fairness of system is evaluated by considering all QoE values f(QoS).

Subjects evaluate test conditions e.g. on a 5-point scale.

QoE Modele.g. f(QoS) = MOS

QoE Fairness

QoS Measurement

QoE Metrics

How to define and calculate QoE fairness?

Which QoE metrics are of interest for providers?

MOS AND QOE

Hoßfeld, Tobias, Poul E. Heegaard, Martín Varela, and Sebastian Möller. "QoE beyond the MOS: an in-depth look at QoE via better metrics and their relation to MOS." Quality and User Experience, no. 1(2) (2016). http://link.springer.com/journal/41233

05/01/2023 4

Quality of Experience

• From Quality of service (QoS) to Quality of Experience (QoE)– QoS: packet loss, delay, jitter, …– QoE: subjective experience/satisfaction of users of a service

• Example: video user interested in video quality andsmooth video playout without interruptions

• QoE model required for evaluation, improving QoE by proper monitoring and management…

O utpu t v ideoN etw ork /Tran sm ission chainim pairm ent s

Inpu t v id eo (k now n re feren ce) E ncoder D ecoder

F RPerceptual

m o del

Q ua lity ra t ing (M O S)

05/01/2023 5

Mean Opinion Score (MOS)

• Mean Opinion Score (MOS): numerical indication of the perceived quality of received media after compression and/or transmission

Excellent

Good

Fair

Poor

Bad

5

4

3

2

1

Imperceptible

Perceptible

Slightly annoying

Annoying

Very annoying

MOS Quality ImpairmentExcellent!

Bad!

Fair!Good!

Poor!

Æ

Fair = 3

05/01/2023 6

Same MOS but Different Distributions

05/01/2023 7

1 2 3 4 50

0.2

0.4

0.6

0.8

1

MOS

prob

abili

ty o

f dis

satis

fied

user

q =3

QoE Beyond the MOS

-20° 80°

On average, it’s fine!

But still in pain!

On average, it’s fine!

Still, some suffer!

MOS: Fair = 3MOS > 3:

75% dissatisfied ! Other metrics!

05/01/2023 8

QoE Metrics

• MOS: average user rating for one test condition• SOS: user diversity for that test condition• Theta-Acceptability: prob. that opinion score is above certain threshold• %GoB: the percentage of users rating Good-or-Better (%GoB)• %PoW: the percentage of users rating Poor-or-Worse (%PoW)• Quantile: user rating of fraction of (satisfied, dissatfied)• Probability distribution: complete

information

• SOS parameter a– quantifies user diversity for one application – relates SOS and MOS

.– SOS parameter is scale independent

Hoßfeld, Tobias, Poul E. Heegaard, Martín Varela, and Sebastian Möller. "QoE beyond the MOS: an in-depth look at QoE via better metrics and their relation to MOS." Quality and User Experience, no. 1(2) (2016). http://link.springer.com/journal/41233

QOE FAIRNESS

Tobias Hoßfeld, Lea Skorin-Kapov, Poul E. Heegaard, Martin Varela. Definition of QoE Fairness in Shared Systems. IEEE Comm. Letters, accepted Oct. 2016

05/01/2023 10

QoE Metrics and QoE Fairness

System

Fairness of system is evaluated by considering all QoE values f(QoS).

Subjects evaluate test conditions e.g. on a 5-point scale.

QoE Modele.g. f(QoS) = MOS

QoE Fairness

QoS Measurement

QoE Metrics

How to define and calculate QoE fairness?

Which QoE metrics are of interest for providers?

05/01/2023 11

QoE Fairness

• How to define and calculate QoE fairness?

System

Fairness of system is evaluated by considering all QoE values f(QoS).

QoE Modele.g. f(QoS) = MOS

QoE Fairness

QoS Measurement

05/01/2023 12

Which system is better?

• A) 10% experience best QoE; 90% worst QoE• B) 90% experience best QoE; 10% worst QoE

QoE on 5-point scale• L=1: worst QoE• H=5: best QoE

Normalized QoE (linear transformation) • L=0: worst QoE• H=1: best QoE

05/01/2023 13

Which system is fairer?

• A) 10% experience best QoE; 90% worst QoE• B) 90% experience best QoE; 10% worst QoE

QoE on 5-point scale• L=1: worst QoE• H=5: best QoE

Normalized QoE (linear transformation) • L=0: worst QoE• H=1: best QoE

05/01/2023 14

For which x is the system maximal unfair?

• In the system– x% of users experience maximum (best) QoE: – 100-x% of users experience minimum (worst) QoE:

Jain’s fairness index

0 0.2 0.4 0.6 0.8 1ratio x of users experiencing maximum QoE H

0

0.2

0.4

0.6

0.8

1

fairn

ess

inde

x

QoE fairness FJain's index J

05/01/2023 15

Desirable Properties of a QoE Fairness Index

• (a) Population size independence• (b) Scale and metric independence• (c) Boundedness [0;1]• (d) Continuity• (e) Intuitive

Jain‘s fairness index designed for those properties

05/01/2023 16

For which x is the system maximal unfair?

• In the system– x% of users experience maximum (best) QoE: – 100-x% of users experience minimum (worst) QoE:

0 0.2 0.4 0.6 0.8 1ratio x of users experiencing maximum QoE H

0

0.2

0.4

0.6

0.8

1

fairn

ess

inde

x

QoE fairness FJain's index J

05/01/2023 17

Desirable Properties of a QoE Fairness Index

• (a) Population size independence• (b) Scale and metric independence• (c) Boundedness [0;1]• (d) Continuity• (e) Intuitive

• (f) Deviation symmetric• (g) QoE level independence• (h) Valid for multi-applications

Jain‘s fairness index designed for those properties

Specific to QoE Fairness

05/01/2023 18

Definition of QoE Fairness Index

• QoE model maps QoS parameters x to QoE in – E.g. is the MOS value on a 5-point scale,

• In a system with users, QoE values are random variable

• Maximum standard deviation of .

• Fairness index

𝐹=1− 𝜎𝜎𝑚𝑎𝑥

=1− 2𝜎𝐻− 𝐿 1 2 3 4 5

avg. QoE

0

0.5

1

1.5

2

max

. val

ue

max. std. dev.

05/01/2023 19

Illustration

𝐹=1− 𝜎𝜎𝑚𝑎𝑥

=1− 2𝜎𝐻− 𝐿

1=L

5=H

2 3 45-point scale

x x x xx xxx xx xx x xx xx xx x

Avg. QoE

𝟐𝝈

x single user experience

05/01/2023 20

Issues with Jain‘s Fairness Index

• Coefficient of variation: only useful for ratio scales– Requires natural zero point– No meaning for data on interval scale

• QoE is given on interval scales– Coefficient of variation is not a valid measurement

1=L

5=H

2 3 45-point scale

x x x xx xxx xx xx x xx xx xx x

05/01/2023 21

Comparison: Jain and QoE Fairness Index

• Jain’s fairness index

• QoE fairness index

Jain’s J violates(f) Deviation symmetric(g) QoE level independence

Jain’s J is not very sensitive.J for max. standard deviation:(for 5-point scale)

05/01/2023 22

Some numbers

Scenario~ Description J F

1 All users experience 1. 1 1

2 50% experience 1 and 50% experience 2. 0.90 0.75

3 50% experience 1 and 50% experience 3. 0.80 0.50

4 50% experience 1 and 50% experience 4. 0.74 0.25

5 50% experience 1 and 50% experience 5. 0.69 0.00

6 50% experience 2 and 50% experience 4. 0.90 0.50

7 50% experience 2.9 and 50% experience 4.9. 0.94 0.50

8 Uniform distribution . 0.75 0.42

05/01/2023 23

QoS Fairness != QoE Fairness

05/01/2023 24

References

• Hoßfeld, Tobias, Poul E. Heegaard, Martín Varela, and Sebastian Möller. "QoE beyond the MOS: an in-depth look at QoE via better metrics and their relation to MOS." Quality and User Experience 1, no. 1 (2016): 2. – Open access: http://link.springer.com/journal/41233 – Scripts: https://github.com/hossfeld/QoE-Metrics/wiki – Formal Definition of QoE Metrics: http://arxiv.org/abs/1607.00321

• Tobias Hoßfeld, Lea Skorin-Kapov, Poul E. Heegaard, Martin Varela. Definition of QoE Fairness in Shared Systems. IEEE Comm. Letters, accepted Oct. 2016– http://ieeexplore.ieee.org/document/7588099/