lecture/lab session 2lecture/lab session 2 inputs and outputs

1

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Lecture/Lab Session 2Lecture/Lab Session 2Inputs and Outputs

May 4 2009May 4, 2009


Outline

ReviewInputs of Encoders: Image/Video FormatsOutputs of Decoders: Perceptual Quality IssueMATLAB Exercises• Reading and showing images and video sequences• Getting familiar with color spaces• Examining spatial and temporal redundancies• Examining spatial and temporal redundancies• Examining psychovisual redundancies• Measuring perceptual quality with PSNR

1


Review


Image and video coding: A big picturePredictive

Coding

Pre-Processing

LossyCoding

Lossless Coding

Post-Processing

InputImage/Video

Encoded

Coding

Predictive

…110

Visual Quality

3

EncodedImage/Video

Post-Processing

LossyCoding

Lossless Coding Pre-Processing

Predictive Coding

DecodedImage/Video

11001…

Visual Quality Measurement

2


Some termsLuminance = Brightness = Lightness = Intensity = ValueHue: Saturation = Colorfulness = Chroma = Vividness = PurityChromaticity = Hue + Saturation => It is a 2-tuple.Chromaticity Diagram: a 2-D diagram showing chromaticitiesColor = Luminance + Chromaticity => So, it is a 3-tuple.Chrominance (Chroma) = Color DifferenceColor Space: a 3-D space of colorsColor Mixing Systems (light) vs. Color Appearance Systems (perception)Gamut: range of colors (in a color space)JND: Just-Noticeable Difference (50% accuracy)

4


HSV/HSL (HSI/HSB) color spaces

5

HSV color space HSL color space


sRGB vs. CIE 1931 XYZ color spaces

6


Inputs of Encoders:Image/Video Formats

3


Color conversion in image/video encoding

Pre-ProcessingInput

Image/Video Color Space ConversionRGB R’G’B’ Y’UV/

Y’PbPr

A/D Conversion

Y’CbCr

g

ChromaSubsampling

8

Other Parts of Encoding ProcessEncoded

Image/Video

…001110101001…


Color conversion: R’G’B’ => Y’CbCrR’,G’,B’∈[0,1], kr, kb∈(0,1), kr+kb<1, , [ , ], r b ( , ), r b

Y’=kr*R’+(1-kr-kb)*G’+kb*B’∈[0,1]Pb=0.5*(B’-Y’)/(1-kb)∈[-0.5,0.5]Pr=0.5*(R’-Y’)/(1-kr)∈[-0.5,0.5]Values of kb and kr can be different!• ITU-T BT.601 (SDTV): kr=0.229, kb=0.114• ITU-T BT.709 (HDTV): k =0.2126, kb=0.0722ITU T BT.709 (HDTV): kr 0.2126, kb 0.0722• ANSI/SMPTE 240M-1995 (HDTV): kr=0.212, kb=0.087

Y’PbPr => Y’CbCr (taking MPEG-2 as an example)• Y’=219*Y’+16∈[16,235]

Cb=224*Pb+128∈[16,240]Cr=224*Pr+128∈[16,240]

9


Chroma subsampling formats

A 4X2 Image

Y Cb CrY Sampling Locations

Cb SamplingLocations

Cr Sampling Locations

4:4:4 4 4 4 ALL ALL ALL

A 4X2 ImageP(1,1) P(1,2) P(1,3) P(1,4)P(2,1) P(2,2) P(2,3) P(2,4)

10

4:2:2 4 2 2 ALL P(1,1) P(1,3)P(2,1) P(2,3)

P(1,1) P(1,3)P(2,1) P(2,3)

4:2:0 4 1 1 ALL P(1,1) P(1,3) P(1,1) P(1,3)4:1:1 4 1 1 ALL P(1,1) P(2,1) P(1,1) P(2,1)


Progressive vs. Interlacing

Progressive mode: each row of a frame is scanned one by g yoneInterlacing mode: a frame is divided into two fields –odd rows are scanned first and even rows later.Benefit: bandwidth saving close to ½• 1920×1080×60Hz interlacing HDTV: 1920/2×1080×60×24=

1492992000≈1.4 G bits/second• 1280×720×60Hz progressive HDTV: 1280×720×60×24=

1327104000≈1.236 G bits/second

Problem:

11

4


Interlacing mode: Frames vs. Fields

A frame may be divided into two fieldsA frame may be divided into two fields• Top filed + Bottom field

12




13




14


Digital image formatsNo standard spatial resolutions• For research purpose: 2n×2n, such as 128×128, 256×256, 512×512,

1024×1024Uncompressed images• BMP (*.bmp): fileheader + infoheader + [palette] + data (row by row,

bottom-up)• Netpbm formats: PBM/PGM/PPM (*.pbm/*.pgm/*.ppm)

Losslessly compressed images• TIFF = Tagged Image File Format (*.tiff/*.tif)• PNG = Portable Network Graphics (*.png)• GIF = Graphics Interchange Format (*.gif) – up to 256 colors

Lossily compressed images• JPEG (*.jpg), JPEG2000 (*.jp2/*.j2k)

15

5


Digital video formats

CIF = Common Intermediate Format (since H.261)( )• CIF (Full CIF = FCIF) = 352×288• QCIF (Quarter CIF) = 176×144• SQCIF (Sub Quarter CIF) = 128×96• 4CIF = 4×CIF = 704×576• 16CIF = 4×4CIF = 1408×1152

SIF = Source Input Format (since MPEG-1)p ( )• 625/50 (TV: PAL/SECAM) = 352×288/360×288• 525/59.94 (TV: NTSC) = 352×240/ 360×240• Sub-SIF (Computers) = 320×288 or 384×288

16


YUV video file format (.yuv/.cif/.qcif/.sif/…)

Planar formatsPlanar formats• YUV = YV12 = I420 = IYUV (4:2:0)• YV16 (4:2:2)• …

Packed formats• UYVY = UYNV = Y422 (4:2:2)UYVY UYNV Y422 (4:2:2)• YUY2 = YUNV = V422 = YUYV (4:2:2)• …

More info available at http://www.fourcc.org/yuv.php

17


YUV4MPEG2 format (.y4m)File Header• File signature: ‘YUV4MPEG2’• Parameters

• Width, height and frame rate: ‘ Wxxx Hyyy Fa:b’• Interlacing: ‘ Ip’ (progressive), ‘ It’ (top field first), ‘ Ib’ (bottom field first), ‘ Im’ (mixed

mode, detailed in frame headers)• Aspect ratio: ‘ Aa:b’• Color space (Chroma format): ‘ C4xx...’• Comment: ‘ X….’

FFrames• Frame Header

• ‘FRAME’ + a number of parameters (optional) + 0x0A

• Frame (YUV planar format)More information is available at http://wiki.multimedia.cx/index.php?title=YUV4MPEG2

18


Multimedia container/wrapper formats

AVI = Audio Video Interleave (*.avi)( )FLV = Flash Video (*.flv)ASF = Advanced Systems Format (*.asf)MPEG-TS (Transport Stream) & MPEG-PS (Program Stream) (*.mpg/*.ts/*.ps)MP4 = MPEG-4 Part 14 (*.mp4) => 3GP (*.3gp/*.3g2)MOV (Quicktime) (*.mov)RealMedia (*.rm/*.rmvb)…

19

6


Outputs of Decoders:Perceptual Quality Issue


Image and video coding: Quality issueQuality M t i

Visual Quality Measurement

Decodedimage/video

Inputimage/video

Meterics

Encoder Decoder

Encodedimage/video

21


Visual quality measurement: Subjective

DSIS (Double Stimulus Impairment Scale)DSIS (Double Stimulus Impairment Scale)DSCQS (Double Stimulus Continuous Quality Scale)SSCQE (Single Stimulus Continuous Quality Evaluation)S t th d h bSome measurement methods have been standardized: ITU-R BT.500, ITU-R BT.710, ITU-T P.910

22


Visual quality measurement: Objective

Two images: the original one f(x,y) and the g g f( ,y)decoded one f’(x,y)MSE = Mean Squared Error

SNR = Signal-to-Noise Ratio

1

MN

Xx,y

(f 0(x, y)− f(x, y))2

Ã P(f 0(x y))2

!PSNR = Peak Signal-to-Noise Ratio10 log10

Ã Px,y(f (x, y))P

x,y(f0(x, y)− f(x, y))2

!

10 log10

ÃL2P

x,y(f0(x, y)− f(x, y))2

!

23

7



PSNR=32.7 dB PSNR=37.5 dBThe original image

Research Question: PSNR is not good enough to perfectly reflect visual quality, more advanced metrics considering HVS are wanted. Some objective metrics are being standardized by the ITU-T.

24



SSIM = Structural Similarity IndexSSIM Structural Similarity Index

VQM = Video Quality MetricMPQM = Moving Pictures Quality MetricNQM = Noise Quality Measure

SSIM =(2E(f(x, y)f 0(x, y)) + C1) (2σf,f 0 + C2)

(E(f(x, y))2 + E(f 0(x, y))2 + C1)³σ2f + σ2f 0 + C2

´

Q Q y

25


MATLAB Exercises


Reading and Showing Image/Video

Read an image.g• f=imread(‘Images/lena_color.bmp’);

Show an image.• imshow(f);

Read a frame from a YUV video.• f=yuvread(‘Video/news.qcif’,1);

Show a video frameShow a video frame.• imshow(ycbcr2rgb(YUV4XX_444(f)));

Play back a YUV video.• yuvplay(‘Video/news.qcif’,‘All’);

27

8


Getting familiar with color spaces

Show YUV planes of a video.pf=yuvread(‘test.cif',1);subplot(2,2,1:2);imshow(f.Y);subplot(2,2,3);imshow(f.U);subplot(2,2,4);imshow(f.V);

Try to add pseudo-colors to U- and V-planes.• Tip: assume Y=0.5 and another chroma channel is 0.

Try to read a RGB image and convert it to YUV color space.

28


YUV video player => Y4M video player

Read the code of YUV video playerRead the code of YUV video player.• Two files: YUVplayer.m and YUVplayer.fig.• Run “guide YUVplayer.fig” to open the second

file.• Read MATLAB help documents to learn how to

design GUI with MATLAB.g

Try to implement a Y4M video player.• This can be a take-home assignment.

29

lecture/lab session 2lecture/lab session 2 inputs and outputs

Documents