Results of the ATIS/T1A1.1Ad Hoc Group on Full-Reference Video

Quality Metrics (FR-VQM)

VSF MeetingOctober 3, 2001

John PearsonSarnoff Corporation

jpearson@sarnoff.com

Take Home Messages

• Tariff’s can now include Visual Quality Metrics (Full Reference)

• The basis for this is a family of 4 Technical Reports by ATIS/T1A1

• The T1A1 approach is extensible to additional Visual Quality Metrics, and does NOT establish a Standard

Outline

• Why is measuring Visual Quality important?

• Why is measuring Visual Quality hard?

• International Standards for VQM’s

• T1A1 Technical Reports

FR-VQM Needs of US Telecom

• Digital video processing can create objectionable noise• End-to-End QoS across the networks of multiple companies

requires agreement on Quality at Transfer Points (Tariffs)• Tariff’s require ANSI sanctioned technical documentation

Site of Video Origination(e.g., Denver)

Transfer Between

Network A & B

Site of Video Consumption(e.g., Mexico City)

Company Auses VQM-A

Company Buses VQM-B

Q-A Q-?? Q-B

Blocky “Digital” NoiseRandom “Analog” Noise

Digital Video Creates “Patterned” Noise... Human visual response to patterned noise highly non-linear ...

MSE = 27.10 MSE = 21.26

Measures like MSE suitable for Analog noise no longer work for Digital noise

Codec Frame

Source Frame

Difference Map

Patterned noise in the sky much more perceptibleeven though much smallerin terms of pixel differences

Visual Quality Metrics... correlate well across scene types, unlike MSE ...

0

1

2

3

4

8 10 12 14 16 18

BusCosbyFlower GardenMobile CalendarNBA

VDM Fidelity Metric

Extremelyperceptible

Clearlyperceptible

Mildlyperceptible

Barelyperceptible

Notperceptible

DS

IS R

ati

ng

Correlation coefficient = 0.96

0

1

2

3

4

0 50 100 150 200 250

BusCosbyFlower GardenMobile CalendarNBA

MSE Fidelity Metric

DS

IS R

atin

g

Correlation coefficient = 0.39

Visual Discrimination Model Mean-Squared Error

Bars show 5% confidence intervals

Mean of 80 trials for 20

subjects

Vital Role of Subjective Database

• Goal of VQMs is to approximate subjective quality assessments (SQA)

• The relevance of the SQA depends on:– Test sequences (SRC’s)– Distortion generators (HRC’s)– Viewing conditions and testing protocols

• Producing a relevant SQA is hard

Three Kinds of VQM’s

• Full Reference (FR)– a double-ended method and is the subject of this Technical Report.

• Reduced Reference (RR)– only reduced video reference information is available. This is also a

double-ended method.

• No Reference (NR)– no reference video signal or information is available. This is a

single-ended method.

• It is generally believed that the FR method will provide the most accurate measurement results while the RR and NR methods will be more convenient for QoS monitoring.

• The T1A1 Technical Reports concern FR methods

R e fe re n c eV id e o

P ro c e s se dV id e o

N o rm a liz e dP ro c e s se d V id e o P ic tu re Q u a li ty

R a tin g (P Q R )

R e p o r te d N o rm a liz a tio nA d ju s tm e n ts

M e a su re m e n tM e th o d

S y s tem U n d e r Te s t

N o rm a liz a tio n

Full-Reference VQM’s with Normalization

International Standards Progress

• VQEG may be several years from recommending a FR-VQM standard to ITU

• Its possible that no single FR-VQM will be a clear “winner”

• The FR-VQM field is young, and significant, steady improvements are expected over the next decade

• It’s possible that several different FR-VQMs may gain industry acceptance

T1A1.1 FR-VQM Strategy… an extensible family of TR’s for FR-VQ, enabling

Industry to move ahead without Standards ...

• Provide guidelines for how Industry can– specify its specific FR-VQM needs– assess the suitability of existing documented FR-VQMs– drive the development by FR-VQM proponents of new/improved

FR-VQM algorithms and products– inter-operate with different FR-VQMs

• Provide guidelines for how FR-VQMs can be– documented in algorithms, accuracy and limitations– quantitatively cross-calibrated to each another

• Extensible framework enabling addition of FR-VQMs– Start by specifying two already disclosed FR-VQMs– Stimulates continued FR-VQM innovation

David Fibush TektronixDick Streeter CBSAlexander Woerner Rohde & SchwarzHarley Myler University of Central FloridaStephen Wolf NTIAStephen Voran NTIAMargaret Pinson NTIAAhmad Ansari SBCPierre Costa SBCDebra Phillips SBCMichael H. Brill Sarnoff CorporationJohn Pearson Sarnoff Corporation (co-chair)Jeffrey Lubin Sarnoff CorporationJohn Grigg Qwest (co-chair)Greg Cermak VerizonPhil Corriveau CRCA. B. Watson NASA Ames Research Center

Primary Contributors

Family of Technical Reports• TR A1: Accuracy and Cross-Calibration (Mike Brill, Sarnoff)

– defines accuracy (statistical analysis), limitations of a FR-VQM

– defines transformation to common scale, for cross-calibration with other applicable FR-VQMs

• TR A2: Normalization Methods (David Fibush, Tektronix)– applied to source and processed video before VQM calculation

– e.g., spatial/temporal registration, gain/level offset calibration, ...

– may utilize special test signals

• TR A3: Peak Signal to Noise Ratio (Steve Wolf, NTIA)– Specify PSNR VQM, following TR A1 and TR A2 guidelines

• TR A4: Objective Perceptual FR-VQM Using a JND-Based Full Reference Technique (David Fibush, Mike Brill)– Specify JND-based FR-VQM, following TR A1 and TR A2 guidelines

• “How to” specify VQM accuracy– with respect to subjective assessments– based on defined statistical analysis

• “How to” specify VQM scope/limitations– type of scene content (“signal”)

• high/low motion, color/b&w, interlaced/progressive

– type/severity of artifacts (“noise”)• e.g., encoding techniques, bit-rates, blurring, blockiness

– subjective testing characteristics• behavior with viewing distance, resolution, gamma, …

• expert vs non-expert viewers

• “How to” cross-calibrate VQMs– determination of mathematical transformation relating one

VQM’s outputs to another’s

TRA1 Defines Basic Methods:

SCOPE

LIMITATIONS

Works well, & has been well tested here

VQEG Database: “SRC’s”

Sequence Characteristics

Baloon-pops film, saturated color, movementNewYork 2 masking effect, movement)

Mobile&Calendar available in both formats, color, movementBetes_pas_betes color, synthetic, movement, scene cut

Le_point color, transparency, movement in all the directionsAutumn_leaves color, landscape, zooming, water fall movement

Football color, movementSailboat almost still

Susie skin colorTempete color, movement

Table B3. Test sequences used to determine test factors, coding technologies and applicationsfor which the PQR method has shown the accuracy specified in section 1.3.4

See the VQEG final report (ITU-T COM9-80, June 2000 – see Annex A) for further detailsregarding the data in these tables. All data is for the 525-line system.

BIT RATE RES METHOD COMMENTS2 Mb/s ¾ resolution mp@ml This is horizontal resolution reduction only2 Mb/s ¾ resolution sp@ml4.5 Mb/s mp@ml With errors3 Mb/s mp@ml With errors4.5 Mb/s mp@ml3 Mb/s mp@ml4.5 Mb/s mp@ml Composite NTSC and/or PAL6 Mb/s mp@ml8 Mb/s mp@ml Composite NTSC and/or PAL8 & 4.5 Mb/s mp@ml Two codecs concatenated19 Mb/s - NTSC-19 Mb/s - NTSC-12 Mb/s

422p@ml NTSC 3 generations

50-50-…-50 Mb/s

422p@ml 7th generation with shift / I frame

19-19-12 Mb/s 422p@ml 3rd generationn/a n/a Multi-generation Betacam with drop-out

compensation (4 or 5, composite/component)Table B1. Test factors, coding technologies and applications for which the PQR method

has shown the accuracy specified in section 1.3.4.

VQEG Database: “HRC’s”

BIT RATE RES METHOD COMMENTS1.5 Mb/s CIF H.263 Full Screen768 kb/s CIF H.263 Full ScreenOther The PQR method specified in this Technical Report is not

appropriate for video conferencing applications that repeatfields or do not meet the latency and delay requirements ofthe video classes. In addition the PQR method is onlyapplicable to typical broadcast transmission systems withvery low error rates such as those included in the VQEGtests.

JND/PQR & PSNR Limitations: no H.263

Contrast Energy Masking

Pyramid Decomposition (4 levels)

Y (from Front-End processing)

Level 0 Level 1 Level 2 Level 3

Spatial Filtering and Contrast Computation

Contrast Energy Masking

Temporal Filtering andContrast Computation

To Chroma Processing

Luma JND Map

Luma Compression

Algorithm Documentation: JND/PQR

Stripping for JND/PQR Registration

Algorithm Documentation: PSNR

21 110

,,ˆ,,1

log20hh

h

vv

v

NO

O

NO

Oprocref

vh

peak

NN

Y

j inn

n

tjiYdtjiY

tPSNR

Table 1. Normalization Requirements for PSNR

Parameter Normalization Tolerance

Luminance gain < 0.2 dB

Luminance DC level < 0.5 % of signal max

Horizontal pixel shift < 0.1 pixel

Vertical line shift < 0.1 line

This tolerance implies field-accurate temporal registration.

Normalization Requirements

Parameter Normalization Tolerance

Luminance level < 0.2 dB of peak white

Color-difference level < 0.2 dB of max allowed excursion

Luminance DC level < 0.5 % of peak white

Color-difference DC level < 0.5% of max allowed excursion

Channel-to-channel delay offset < 2 ns

Horizontal pixel shift < 0.1 pixel

Vertical line shift 0 lines (limited to integer line shifts)

Temporal shift 0 fields

Table 1. Normalization parameters and tolerance

JND/PQR

PSNR

VQEG data & Logistic-mapped PQR

(10)

-

10

20

30

40

50

60

- 2 4 6 8 10 12

PQR

MDO

S VQEG DataLogistic

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10 12 14

Native PQR values

Co

mm

on

VQ

M S

cal

e

Logistic-mapped PQR for Common Scale… provides approach for cross-calibration...

Accuracy -- 3 Methods

• RMSE

• Resolving Power

• Classification of Errors

Confidence vs.D-VQM: JND/PQR

Confidence vs.D-VQM: PSNR

RMSE• RMSE: root mean square error between

subjective and objective normalized scores

A first order calculation of resolving power can be made by simply calculating the root meansquare error (RMSE) of the subjective scores versus the objective values in the normalizeddomain. Differences in VQM values equal to the RMSE provide a 68% confidence level and 1.96times the RMSE provides a 95% confidence level. While this method does not give the sameresult as the more complex approach it is easily understood and may be quite useful consideringthe accuracy levels in operational environments.

VQM_RMSE = 0.06723

This corresponds approximately to the more accurate curve of figure 4 as shown below.

Confidence level Figure 4 Per RMSE

68% 0.053 0.066

95% 0.187 0.132

Classification of Errors

Subjective Score Diffs.

Δo Δo

Δz

Δz

VQM Differences

Bo Eo Wo

Bs

Es

Ws

0

0

Bs Es WsWo False Ranking False Differentiation Correct DecisionEo False Tie Correct Decision False TieBo Correct Decision False Differentiation False Ranking

Bs Es WsWo False Ranking False Differentiation Correct DecisionEo False Tie Correct Decision False TieBo Correct Decision False Differentiation False Ranking

Progress

• T1A1.1 Ad Hoc Group created Feb. 2001, co-chairs John Grigg, John Pearson

• Mail Ballot Approval August 2001• Approved by T1A1.1 25 September 2001• Approved at Plenary meeting of T1A1,

28 September 2001