on qoe metrics and qoe fairness for network & traffic management
TRANSCRIPT
Prof. Dr. Tobias Hoßfeld
Chair of Modeling of Adaptive Systems (MAS)Institute for Computer Science and Business Information Systems (ICB)University of Duisburg-Essen
www.mas.wiwi.uni-due.de
On QoE Metrics and QoE Fairness for Network & Traffic Management
Tobias HossfeldLea Skorin-KapovPoul Heegaard
05/01/2023 2
QoE Metrics and QoE Fairness
System
Fairness of system is evaluated by considering all QoE values f(QoS).
Subjects evaluate test conditions e.g. on a 5-point scale.
QoE Modele.g. f(QoS) = MOS
QoE Fairness
QoS Measurement
QoE Metrics
How to define and calculate QoE fairness?
Which QoE metrics are of interest for providers?
MOS AND QOE
Hoßfeld, Tobias, Poul E. Heegaard, Martín Varela, and Sebastian Möller. "QoE beyond the MOS: an in-depth look at QoE via better metrics and their relation to MOS." Quality and User Experience, no. 1(2) (2016). http://link.springer.com/journal/41233
05/01/2023 4
Quality of Experience
• From Quality of service (QoS) to Quality of Experience (QoE)– QoS: packet loss, delay, jitter, …– QoE: subjective experience/satisfaction of users of a service
• Example: video user interested in video quality andsmooth video playout without interruptions
• QoE model required for evaluation, improving QoE by proper monitoring and management…
O utpu t v ideoN etw ork /Tran sm ission chainim pairm ent s
Inpu t v id eo (k now n re feren ce) E ncoder D ecoder
F RPerceptual
m o del
Q ua lity ra t ing (M O S)
05/01/2023 5
Mean Opinion Score (MOS)
• Mean Opinion Score (MOS): numerical indication of the perceived quality of received media after compression and/or transmission
Excellent
Good
Fair
Poor
Bad
5
4
3
2
1
Imperceptible
Perceptible
Slightly annoying
Annoying
Very annoying
MOS Quality ImpairmentExcellent!
Bad!
Fair!Good!
Poor!
Æ
Fair = 3
05/01/2023 6
Same MOS but Different Distributions
05/01/2023 7
1 2 3 4 50
0.2
0.4
0.6
0.8
1
MOS
prob
abili
ty o
f dis
satis
fied
user
q =3
QoE Beyond the MOS
-20° 80°
On average, it’s fine!
But still in pain!
On average, it’s fine!
Still, some suffer!
MOS: Fair = 3MOS > 3:
75% dissatisfied ! Other metrics!
05/01/2023 8
QoE Metrics
• MOS: average user rating for one test condition• SOS: user diversity for that test condition• Theta-Acceptability: prob. that opinion score is above certain threshold• %GoB: the percentage of users rating Good-or-Better (%GoB)• %PoW: the percentage of users rating Poor-or-Worse (%PoW)• Quantile: user rating of fraction of (satisfied, dissatfied)• Probability distribution: complete
information
• SOS parameter a– quantifies user diversity for one application – relates SOS and MOS
.– SOS parameter is scale independent
Hoßfeld, Tobias, Poul E. Heegaard, Martín Varela, and Sebastian Möller. "QoE beyond the MOS: an in-depth look at QoE via better metrics and their relation to MOS." Quality and User Experience, no. 1(2) (2016). http://link.springer.com/journal/41233
QOE FAIRNESS
Tobias Hoßfeld, Lea Skorin-Kapov, Poul E. Heegaard, Martin Varela. Definition of QoE Fairness in Shared Systems. IEEE Comm. Letters, accepted Oct. 2016
05/01/2023 10
QoE Metrics and QoE Fairness
System
Fairness of system is evaluated by considering all QoE values f(QoS).
Subjects evaluate test conditions e.g. on a 5-point scale.
QoE Modele.g. f(QoS) = MOS
QoE Fairness
QoS Measurement
QoE Metrics
How to define and calculate QoE fairness?
Which QoE metrics are of interest for providers?
05/01/2023 11
QoE Fairness
• How to define and calculate QoE fairness?
System
Fairness of system is evaluated by considering all QoE values f(QoS).
QoE Modele.g. f(QoS) = MOS
QoE Fairness
QoS Measurement
05/01/2023 12
Which system is better?
• A) 10% experience best QoE; 90% worst QoE• B) 90% experience best QoE; 10% worst QoE
QoE on 5-point scale• L=1: worst QoE• H=5: best QoE
Normalized QoE (linear transformation) • L=0: worst QoE• H=1: best QoE
05/01/2023 13
Which system is fairer?
• A) 10% experience best QoE; 90% worst QoE• B) 90% experience best QoE; 10% worst QoE
QoE on 5-point scale• L=1: worst QoE• H=5: best QoE
Normalized QoE (linear transformation) • L=0: worst QoE• H=1: best QoE
05/01/2023 14
For which x is the system maximal unfair?
• In the system– x% of users experience maximum (best) QoE: – 100-x% of users experience minimum (worst) QoE:
Jain’s fairness index
0 0.2 0.4 0.6 0.8 1ratio x of users experiencing maximum QoE H
0
0.2
0.4
0.6
0.8
1
fairn
ess
inde
x
QoE fairness FJain's index J
05/01/2023 15
Desirable Properties of a QoE Fairness Index
• (a) Population size independence• (b) Scale and metric independence• (c) Boundedness [0;1]• (d) Continuity• (e) Intuitive
Jain‘s fairness index designed for those properties
05/01/2023 16
For which x is the system maximal unfair?
• In the system– x% of users experience maximum (best) QoE: – 100-x% of users experience minimum (worst) QoE:
0 0.2 0.4 0.6 0.8 1ratio x of users experiencing maximum QoE H
0
0.2
0.4
0.6
0.8
1
fairn
ess
inde
x
QoE fairness FJain's index J
05/01/2023 17
Desirable Properties of a QoE Fairness Index
• (a) Population size independence• (b) Scale and metric independence• (c) Boundedness [0;1]• (d) Continuity• (e) Intuitive
• (f) Deviation symmetric• (g) QoE level independence• (h) Valid for multi-applications
Jain‘s fairness index designed for those properties
Specific to QoE Fairness
05/01/2023 18
Definition of QoE Fairness Index
• QoE model maps QoS parameters x to QoE in – E.g. is the MOS value on a 5-point scale,
• In a system with users, QoE values are random variable
• Maximum standard deviation of .
• Fairness index
𝐹=1− 𝜎𝜎𝑚𝑎𝑥
=1− 2𝜎𝐻− 𝐿 1 2 3 4 5
avg. QoE
0
0.5
1
1.5
2
max
. val
ue
max. std. dev.
05/01/2023 19
Illustration
𝐹=1− 𝜎𝜎𝑚𝑎𝑥
=1− 2𝜎𝐻− 𝐿
1=L
5=H
2 3 45-point scale
x x x xx xxx xx xx x xx xx xx x
Avg. QoE
𝟐𝝈
x single user experience
05/01/2023 20
Issues with Jain‘s Fairness Index
• Coefficient of variation: only useful for ratio scales– Requires natural zero point– No meaning for data on interval scale
• QoE is given on interval scales– Coefficient of variation is not a valid measurement
1=L
5=H
2 3 45-point scale
x x x xx xxx xx xx x xx xx xx x
05/01/2023 21
Comparison: Jain and QoE Fairness Index
• Jain’s fairness index
• QoE fairness index
Jain’s J violates(f) Deviation symmetric(g) QoE level independence
Jain’s J is not very sensitive.J for max. standard deviation:(for 5-point scale)
05/01/2023 22
Some numbers
Scenario~ Description J F
1 All users experience 1. 1 1
2 50% experience 1 and 50% experience 2. 0.90 0.75
3 50% experience 1 and 50% experience 3. 0.80 0.50
4 50% experience 1 and 50% experience 4. 0.74 0.25
5 50% experience 1 and 50% experience 5. 0.69 0.00
6 50% experience 2 and 50% experience 4. 0.90 0.50
7 50% experience 2.9 and 50% experience 4.9. 0.94 0.50
8 Uniform distribution . 0.75 0.42
05/01/2023 23
QoS Fairness != QoE Fairness
05/01/2023 24
References
• Hoßfeld, Tobias, Poul E. Heegaard, Martín Varela, and Sebastian Möller. "QoE beyond the MOS: an in-depth look at QoE via better metrics and their relation to MOS." Quality and User Experience 1, no. 1 (2016): 2. – Open access: http://link.springer.com/journal/41233 – Scripts: https://github.com/hossfeld/QoE-Metrics/wiki – Formal Definition of QoE Metrics: http://arxiv.org/abs/1607.00321
• Tobias Hoßfeld, Lea Skorin-Kapov, Poul E. Heegaard, Martin Varela. Definition of QoE Fairness in Shared Systems. IEEE Comm. Letters, accepted Oct. 2016– http://ieeexplore.ieee.org/document/7588099/