large-scale video compression research work on quality of experience (qoe) evaluation for vqeg...

7/31/2019 Large-Scale Video Compression Research Work on Quality of Experience (QoE) Evaluation for VQEG JEG-Hybrid Project

1/16

Large-Scale Video Compression Research Work

on Quality of Experience (QoE) Evaluation forVQEG JEG-Hybrid Project

Mikoaj Leszczuk

1

Andrzej Gowacz

1

Jan Derkacz

1

Andrzej Dziech1 Piotr Romaniak2 Bazej Szczerba2

Marcus Barkowsky3

1AGH University, Department of Telecommunications

2AGH University, ACC CYFRONET

3LUNAM Universite, Universite de Nantes

July 04, 2012

1/16
http://goforward/http://find/http://goback/


2/16

Video Quality of Experience (QoE) vs. Video

Transmission Chain

1 Original video acquisition

2 Encoder & streaming server

3 Network

4 User equipment

2/16
http://find/


3/16

Video Quality Experts Group (VQEG)

Vision:To advance the field of video

quality assessment...Main activities today:

Validation of models

Development of objective

models

Improvement of subjective

testing methods

VQEG brings internationalexperts together:

Industry

Academia

Government Organizations

ITU InternationalTelecommunication Union

SDOs Other Standard-

-Developing Organizations

VQEG submits reports to

SDOs, in particular ITU

Reference:

http://www.vqeg.org/

3/16
http://www.vqeg.org/http://www.vqeg.org/http://find/


4/16

Full vs. No Reference Metrics

Full Reference (FR)

Already accurate, but not-applicableon the client side

No Reference (NR)

Still inaccurate, but applicableon the client side

Diagram images borrowed from http://mmspg.epfl.ch/page-58337-en.html

4/16
http://mmspg.epfl.ch/page-58337-en.htmlhttp://mmspg.epfl.ch/page-58337-en.htmlhttp://find/


5/16

Competitive vs. Collaborative Approach

Competitive Approach

ITU Recommendation

Winning Model

VQEG Test via

Competitive Project

Contributed

Model

.1

...Contributed

Model

.N

Collaborative Approach

ITU Recommendation

Joint (Super-)Model

VQEG Test via

Collaborative (JEG) Project

Contributed

Model

.1

...Contributed

Model

.N

5/16
http://find/


6/16

VQEGs Joint Effort Group (JEG) Hybrid Project

Open collaboration working together to develop robust Hybrid

Perceptual/Bit-Stream model

Developed and made available routines to create and capture

bit-stream data and parse bit-streamsEfforts are under-way into developing subjectively rated video quality

datasets with bit-stream data that can be used by all researchers

Goal: to produce one model that combines metrics developed

separately by variety of researchers

6/16
http://find/


7/16

What about Ground Truth?

Classical Methodology: ITU-T P.910s Absolute Category RatingThe typical approach to learning about video quality involves

conducting subjective experiments

They are time- and resources-consuming

While 24 subjects are enough to test a small set of parameters, it is

often necessary to use up to 100 testers for more parameters

Tests using widely accepted ITU-Ts methodology, involve screening

video sequences and collecting of quality marks, one by one

7/16
http://find/http://goback/


8/16

Multidimensional Problem

Several dimensions to be traversed:

Several Source Reference Channels/Circuits (SRCs)

Several encoding parametersSeveral networking parameters

... and we regret to inform you that cutting multidimensional corners

through hyperspace, works in Science Fiction only ;-(

Image courtesy of Galactic Empire (Star Wars) ;-)

8/16

N l


9/16

Novelty

Main innovation:comprehensive, large-scale

evaluation of video QoE,

based on information

provided by FR metrics

FR metrics:PSNR: Peak S./N. Ratio

SSIM: Structural Sim. Index

VQM: Video Quality Metric

SwissQuals Metric

TetraVQM

VIF: Visual Inform. Fidelity

MOVIE: MOtion-based

Video Integrity Evaluation

Support by indicators:Blockiness

Blurriness

Exposure

Noisiness

Flickering

Spatial Activity (Intensity)

Temporal Activity (Intensity)

Spatial Correlation

Energy

Homogeneity

VarianceContrast

Colour Layout Descriptor

Edge Histogram Descriptor

9/16

Vid Q lit f E i (Q E) Vid


10/16

Video Quality of Experience (QoE) vs. Video

Transmission Chain Contribution

1 Original video acquisition

2 Encoder& streaming server

3 Network

4 User equipment

10/16

C ll ti f S R f Ch l /Ci it


11/16

Collections of Source Reference Channels/Circuits

(SRCs)

JEG has a dataset that can be used

Many sequences originate from the database of the Consumer

Digital Video Library (CDVL)

Example Test Sequences

11/16

H th ti l R f Ci it (HRC) P t S t


12/16

Hypothetical Reference Circuit (HRC) Parameter Sets

Bit-rate/quantization factor, with at least 5+5 choices

Different Group of Pictures (GoP) sizes and structures:

2 choices of I frame frequency

2 choices of B/P frames frequency2 different settings of Hierarchical coding

Structure of slices of at least 2 different lengths

Number of frames per second original and halved

Resolution original and halved

12/16

Experiment Scale


13/16

Experiment Scale

Each SRC needs to be:

10 seconds long

With 30 FPS

With resolution ranging:

From SD (720 480/576)

To HD (1920 1080)

Each video frame needs to beencoded

1 second per video frame has

been evaluated to be good

estimate for using JM

Reference Software usingsingle CPU core

For each video frame, video

quality metrics need to be

calculated

For each video frame, videocontent characteristics need to

be calculated

13/16

Calculation Time using Single Core


14/16

Calculation Time using Single Core

Calculation time using a single core has been estimated as approx. 889

days

Assumption 1: 19, 200, 000 video frames.

Assumption 2: compression of a single video frame in 1 s.Assumption 3: three video quality metrics to be calculated for all video

frames.

Assumption 4: calculation of a single video quality metric for one frame

in approx. 1 s.

14/16

Conclusions


15/16

Conclusions

Result in creation of very largecollection of test sequencesthat can be used in:

Subjective tests

Objective tests

Development of algorithms

for video encoding andobjective-perceptual quality

estimation

Codec optimisation

Resulting data to be available

to community as it may beuseful in other projects as well

Computer cluster used to

accelerate calculations

Zeus supercomputer:At AGH University Computer

Centre CYFRONET

Most efficient computer in

Poland (185 TFLOPS)

15/16

Acknowledgements


16/16

Acknowledgements

Work financed by The Polish National Centre for Research and

Development (NCBiR) as part of project no. SP/I/1/77065/10 (SYNAT)

Previous slide image source: Wikipedia

16/16
http://find/

large-scale video compression research work on quality of experience (qoe) evaluation for vqeg...

Documents