Improved input-state linearization in video bitrate controllers

Noam Korem

Presentation outline

Video encoding and rate control Classic models Suggested improvement Simulation results Summary and conclusions

• Video encoding and rate control• Classic R-Q models• Suggested improvement• Simulation results• Summary / Conclusions

Generic video encoder*

I P

i-2 i-1

P

i

P

i+1

DCT

VLC..01001..

qi-2

+-

Iqi-

2

IDCT DCT

VLC..01001..

qi-1 Iqi-

1

IDCT

+

+0

+

+

Iqi

IDCT

+

+

+-

DCT

qi

VLC..01001..

-+

DCT

qi+

1

VLC..01001..

Intra Predicted

Discrete CosineTransform

• Video encoding and rate control• Classic R-Q models• Suggested improvement• Simulation results• Summary / Conclusions

Discrete Cosine Transform (DCT) Resembles Discrete Fourier Transform Purely real (real transform to real) Allows representation in the frequency domain, usually more

compact

1

0

1

0

1

0

1

0

2

12cos

2

12cos,

2,

~

2

12cos

2

12cos,

2,

N

u

N

v

N

x

N

y

N

vy

N

uxvuFvCuC

Nyxf

N

vy

N

uxyxfvCuC

NvuF

Spatial domain Frequency domain

Quantization

Quantization scale = 1 Quantization scale = 31

Generic video encoder with rate control mechanism

Frameencoder

bitstargetbit rate

Non-linear Bit ratecontroller

qi

Si

Linearcontroller

qi=Q(Ti,..)Ti

Non linear bitrate controller(Input-state linearization)

The problem

P

i-1

P

i

Iqi-

1

IDCT

+

+

Iqi

IDCT

+

+

+-

DCT

qi

VLC..01001..

-+

DCT

qi+

1

VLC..01001..

Energy of difference frame i is dependent on the q(i-1)

Coded information of frame iis dependent on q(i), q(i-1)• Video encoding and

rate control √• Classic R-Q models• Suggested improvement• Simulation results• Summary / Conclusions

The problem, cont.

Classic Rate-Quantization models do not depend on reference frame quantization scale

22

11

1

11

:

:5

:5

iii

iii

iii

QQRQuadratic

QXRTMdGeneralize

QXRTM

Improved R-Q model

i-1

i-1(q)

i

Qi-1

ΔQi-1

ΔRi

Qi

1ii

ii

QR

QR

Qi-1• Video encoding and rate control √• Classic R-Q models √• Suggested improvement• Simulation results• Summary / Conclusions

Previous frame quantizerdependency

Ri(Qi)

0

20000

40000

60000

80000

100000

120000

140000

160000

180000

1 6 11 16 21 26 31Qi

bit

s

Ri(Qi-1)

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

1 6 11 16 21 26 31Qi-1

bit

s

Ri(Qi-1), Qi=8Ri(Qi)

Improved R-Q model, cont.

iiiiii QQYQRQQR 111,~

iii RSe

01

0

11

1

11

iiii

iii

iii

i QQQQ

RSY

QQYY

Simulations

Encode real videos, compare accuracy of traditional model (TM5) against improved model

ffmpeg open-source video encoder with enhanced quantization and bit rate control used as encoding model (MPEG4).

All test videos are YUV 4:2:0, QVGA (320x240), encoded at 370kbps (74CR)

• Video encoding and rate control √• Classic R-Q models √• Suggested improvement √• Simulation results• Summary / Conclusions

1111

iiii

ii QSX

Q

XRTM5 model:

Simulations

Absolute prediction error is calculated for each frame, for both models

Delta of absolute prediction errors is used as performance measurement

i

ii

S

RSabsErr

iii absErrTM5ovedabsErrImpr absErr

Simulation results

Carosel

-10

-8

-6

-4

-2

0

2

4

6

8

10

1 21 41 61 81 101 121 141 161 181 201 221 241 261 281

Frame

Del

ta o

f ab

solu

te

pre

dic

tio

n e

rro

r Mean Abs Err*TM5: 7.18%TM5i: 4.8%(33% improve)

Opera singer

-8

-6

-4

-2

0

2

4

6

8

10

1 21 41 61 81 101 121 141 161 181 201 221 241 261 281

FrameD

elta

of

abso

lute

p

red

icti

on

err

or

Mean Abs Err*TM5: 17.07%TM5i: 15.03%(12% improve)

Simulation results, cont.Car race

-15

-10

-5

0

5

10

15

1 30 59 88 117 146 175 204 233 262 291 320 349 378 407 436

FrameD

elta

of

abso

lute

p

red

icti

on

err

or

Mean Abs Err*TM5: 22.3%TM5i: 19%(14.7% improve)

Swings

-15

-10

-5

0

5

10

15

1 44 87 130 173 216 259 302 345 388 431 474 517 560

Frame

De

lta

of

ab

so

lute

p

red

icti

on

err

or Mean Abs Err*

TM5: 16.7%TM5i: 14.1%(15.6% improve)

Simulation results, cont.

Lady with phone

-20

-15

-10

-5

0

5

10

15

1 22 43 64 85 106 127 148 169 190 211 232 253 274 295

Frame

De

lta

of

ab

so

lute

p

red

icti

on

err

or

Mean Abs Err*TM5: 34.2%TM5i: 32.2%(6% improve)

Simulation results, degradation

Pearl Harbor

-20

-15

-10

-5

0

5

10

15

20

1 32 63 94 125 156 187 218 249 280 311 342 373 404 435

Frame

De

lta

of

ab

so

lute

p

red

icti

on

err

or

Mean Abs Err*TM5: 13.1%TM5i: 13.2%(1% degrade)

Summary & Conclusions

Video encoding with rate controller scheme presented (camcorders, live streaming)

Video rate and distortion depend on quantization of reference frame, classic models ignore.

Improved R-Q model allows more accurate input-state linearization, demands more calculations

Improved R-Q model does not deliver when scene complexity changes rapidly

• Video encoding and rate control √• Classic R-Q models √• Suggested improvement √• Simulation results √• Summary / Conclusions

Future work Prediction quality degrades due to input noise –

changes in scene complexity More accurate complexity estimation (more

accurate prediction for Xi) based on: frame content motion estimation

would improve R-Q model accuracy Try the model improvement on other R-Q

models A complete model will include past quantization

parameters Ri=Ri(Qi,Qi-1,Qi-2,…)

Related work

Video Group, "Test Model 5" JTC1/SC29/WG11 Coding of Moving Pictures and associated Audio MPEG 1994, section 10

Tihao Chiang and Ya-Qin Zhang "A New Rate Control Scheme Using Quadratic Rate Distortion Model", IEEE International Conference on Image Processing, 1996.   10.1109/ICIP.1996.560604

Liang-Jin Lin; Ortega, A. “Bit-rate control using piecewise approximated rate-distortion characteristics” Circuits and Systems for Video Technology, IEEE Transactions on, Volume 8, Issue 4, Aug 1998 Page(s):446 - 459 10.1109/76.709411

Saw, Y.-S.; Grant, P.M.; Hannah, J.M., "Rate-distortion analysis of nonlinear quantisers for MPEG videocoders: sigmoidal and unimodal quantiser control functions" Vision, Image and Signal Processing, IEE Proceedings- Volume 145, Issue 4, Aug 1998 Page(s):249 – 256

Ma, S.; Wen Gao; Yan Lu; "Rate-distortion analysis for H.264/AVC video coding and its application to rate control“ Circuits and Systems for Video Technology, IEEE Transactions on Volume 15,  Issue 12,  Dec. 2005 Page(s):1533 - 1544 10.1109/TCSVT.2005.857300

Many thanks:

Shai Mazor

Orly Wigderson

Kobi Kohai

For the guidance, patience and assistance on all aspects of the project