Adaptive Content-Aware Scaling for Improved Video Streaming.Avanish TripathiMark ClaypoolPresented by: Huahui Wu

Worcester Polytechnic Institute http://www.cs.wpi.edu/~claypool

MotivationInternet is more useful with MultimediaTCP is the de facto standard… but

It is not ideal for multimedia

UDP can send with constant bit rate It doesn’t have congestion control

Other network protocols e.g. TFRC, RAP Ideal-rate based with smooth rates But the sender treats all the data same.

Motivation

Random packet drop by routers during congestion decreases qualityApplication know best data to send for a given rateNeed application level solution…

…Media ScalingNeed media scaling: Application level data-rate reductionScaling types: Temporal scaling Quality scaling

Best scaling type depends on motion“Content of the stream should influence the choice of scaling mechanism”To the best of our knowledge this idea has not yet been completely evaluated

Methodology: Content-Aware Scaling

Develop and verify motion measurement mechanismDefine temporal and quality scaling levelsEvaluate the potential impact of content-aware scaling (User Study 1)Build system to do content-aware scaling adaptivelyEvaluate the practical impact of the full system (User Study 2)

Motion MeasurementMPEG has three kinds of frames I- Intra encoding P- Predictive encoding B- Bi-directional predictive

encoding

Subdivided into Macroblocks Intra macroblocks Predictive macroblocks Interpolated macroblocks

More when adjacent frames are similar

Motion MeasurementOur hypothesis High percent of interpolated macroblocks means

low motion Low percent of interpolated macroblocks means

high motion

Conducted a pilot study to test our hypothesis 18 Video clips, one scene each (5~10s) Visually divide frame into 16 sub-blocks Manually count number of blocks that have motion Correlate with percent of Interpolated macroblocks.

Pilot Study Result: Motion Measurement

Methodology: Content-Aware Scaling

Develop and verify motion measurement mechanismDefine temporal and quality scaling levelsEvaluate the potential impact of content-aware scaling (User Study 1)Build system to do content-aware scaling adaptivelyEvaluate the practical impact of the full system (User Study 2)

Filtering

Extend a system developed at Lancaster university

Frame dropping filter (Temporal Scaling) Requantization filter (Quality Scaling)

Scale level table 1

Type Level Method Fps Bwidth(%)None N/A N/A 30 100Temporal 1 No B 13 70Temporal 2 No P or B 5 11Quality 1 Q = 7 30 65Quality 2 Q = 31 30 10

Methodology: Content-Aware Scaling

Develop and verify motion measurement mechanismDefine temporal and quality scaling levelsEvaluate the potential impact of content-aware scaling (User Study 1)Build system to do content-aware scaling adaptivelyEvaluate the practical impact of the full system (User Study 2)

User Study 122 graduate and undergraduate students in the departmentPlatform: 3 Pentium III machines with 128MB RAM

running Linux Clips were on local hard drives

Eighteen clips, 10 second each Some high motion, some low motion

Five versions of each clip: Perfect, Temporal Level 1, Temporal Level 2,

Quality Level 1, Quality Level 2

User Study 1

• Users rated the clips with numbers from 0 -100

Results Four men sitting at a bar

Low Motion ( 70% interpolated macroblocks)

ResultsA man on horseback tries to rope a bull

High Motion (27% interpolated macroblocks)

ResultsTemporal scaling is better for low motion Since the frame changes a little, not all the

frames are important.

Quality scaling is better for high motion Since the frame changes frequently, user

need every frame to catch the motion.

Will a video keep low motion or high motion? No!

Methodology: Content-Aware Scaling

Develop and verify motion measurement mechanismDefine temporal and quality scaling levelsEvaluate the potential impact of content-aware scaling (User Study 1)Build system to do adaptive content-aware scalingEvaluate the practical impact of the full system (User Study 2)

Full System Architecture

MPEG Server

Input

MotionMeasurement

High

Low

Temporal Filter

Quality Filter

Internet Feedback Generator Client

System Functionality

Server is capable of classifying motion as the movie playsThe system is adaptive and scales movies in real-time depending on the loss pattern as received from the feedback moduleTwo more scale levels for finer granularity

Methodology: Content-Aware Scaling

Develop and verify motion measurement mechanismDefine temporal and quality scaling levelsEvaluate the potential impact of content-aware scaling (User Study 1)Build system to do content-aware scaling adaptivelyEvaluate the practical impact of the full system (User Study 2)

User Study 2 Four clips (2 or more scenes) ~30 seconds Four versions of each video clip Perfect Quality Temporal scaling only Quality scaling only Adaptive scaling

Bandwidth distribution functions: how often the rate changes Every 3 seconds (slow) Every 200ms (frequently)

ResultTalk show followed by a car commercial Low motion to High motion Bandwidth changes every 3 seconds

80

59.15 52.95

69.35

0

20

40

60

80

100

None Quality Temporal ACA

Scalling Mechanisms

Per

cept

ual Q

ualit

y

ConclusionsBuild application level solution to the problem of congestion due to unresponsive video streamsDeveloped a mechanism to classify the motion in a video streamShown that content aware scaling can improve user perceived quality by as much as 30%Developed a system to do adaptive content-aware scaling and proved it with user study

Future WorkMore accurate bandwidth distribution function to test our system in the real world. More scaling level to get smoother data rate.Hybrid scaling methods (Quality + Temporal)Audio Scaling

Adaptive Content-Aware Scaling for Improved Video Streaming.Avanish TripathiMark ClaypoolPresented by: Huahui Wu

Worcester Polytechnic Institutehttp://www.cs.wpi.edu/~claypool