levelup: a thin-cloud approach to

LevelUp: A Thin-cloud Approach toGame Livestreaming

Landon Cox (Microsoft Research) Lixiang Ao (UC San Diego)

Game Livestreaming

• Twitch• Average >2m concurrent viewers and >90k concurrent channels• >65m hours streamed and >1.5b hours watched per month

• Market size $40b, expected to grow 18% per year

Game Livestreaming

Cloud-based Video Transcoding Is Expensive

• $300+ to transcode 100 hours of video on Azure• <$20 to livestream 100 hours of single-bitrate video on Wowza

• Reason: Video transcoding is resource demanding, usually requires hardware accelerators in the cloud

Mobile Hardware Trends

GPU video codec ML

• The edge is ready to play a more central role in video livestreaming

Overview of the paper• Idea

• Offload cloud-based transcoding by encoding multi-bitrate videos on broadcasters’smartphones

• In case of bandwidth constraints, viewers boost reduced-resolution video quality withsuper-resolution using ML

• Adopt game-specific CNN models to improve quality• Contributions

• Implementation of LevelUp prototype• Evaluation on COTS mobile devices

• Key Results• 88% improvement for low-resolution game streams• Only 5% battery usage for 30 mins video

LevelUp Design

LevelUp Design – bandwidth constrained

+

Capture screen

Resize to 3 resolutions

Broadcaster’s video pipeline

Upload to server

Encode SegmentsSend to HW

encoder queue

Viewer’s video pipeline – without SR

Download video segment

Decode

Display

Viewer’s video pipeline – with SR

Separate grayscale, chroma

Merge, display

Downloadvideo segment

Decode

SR

Bicubic interpolate

How it builds on previous works

• Incremental work on DNN-based video super resolution (e.g., NAS)

• Similar to Dejavu which is used for video conferencing

• Orthogonal to other advances in codecs, compression, bitrate adaption

Game stream super-resolution

100%90%80%70%60%50%40%30%20%10%

0%

CSGO Dota 2 Fallout 4 GTAV Rust Starcraft Witcher 3 2

Perc

entc

hang

e

PSNR SSIM VMAF

100%90%80%70%60%50%40%30%20%10%

0%


Perc

entc

hang

e

PSNR SSIM VMAF

480p + SR 270p + SR

PSNR: per-pixel, SSIM: structural,VMAF[1]: ML model of human perception

[1] https://github.com/Netflix/vmaf

# of improvements with SR


100%90%80%70%60%50%40%30%20%10%

0%


Perc

entc

hang

e

PSNR SSIM VMAF

100%90%80%70%60%50%40%30%20%10%

0%


Perc

entc

hang

e

PSNR SSIM VMAF

VMAF improved 15-20% VMAF improved 61-88%

[1] https://github.com/Netflix/vmaf

480p + SR 270p + SR


Interpolated, 480x270 LevelUp, 480x270

Quality vs. Bitrate

100

90

80

70

60

50

40

30

20

10

00.1 1 10Bitrate (Mbs)

100

VMA

F

GTAV GTAV (SR)

CSGOCSGO (SR)

Starcraft Starcraft (SR)

Dota 2Dota 2 (SR)

RustRust (SR)

Witcher 3 Witcher 3 (SR)

Fallout 4Fallout 4 (SR)

Quality vs. Bitrate

40

30

20

10

0

50

100

90

80

70

60

0.1 1 10Bitrate (Mbs)

100

VMA

F

GTAV GTAV (SR)

CSGOCSGO (SR)


Dota 2Dota 2 (SR)

RustRust (SR)



VMAF boost by LevelUp

Quality vs. Bitrate

20

10

0

30

40

60

50

70

80

100

90

0.1 1 10 100Bitrate (Mbs)

VMA

F

GTAV GTAV (SR)

CSGOCSGO (SR)


Dota 2Dota 2 (SR)

RustRust (SR)



Low+SR vs. Medium Bitrate:70% lower bandwidth, 3-20% VMAF loss

Medium+SR vs. High Bitrate:90% lower bandwidth, 10-20% VMAF loss

Super-resolution can greatly improve game stream quality w/o extra bandwidth consumption

Can broadcaster encode multi-bitrate streams?• Encode 3 2-second segments (1080p, 480p, 270p) at the same time• Test on iPhone 11 Pro, the iPhone Xs, iPhone 8, iPhone 7, and iPhone

6s• All devices can encode in realtime

• Multi-bitrate encoding is feasible on modern smartphones

Can viewers super-resolve video?

0

0.02

0.2

0.18

0.16

0.14

0.12

0.1

0.08

0.06

0.04

iPhone 11 (A13)

iPhone Xs (A12)

iPhone 8 (A11)

iPhone 7 (A10)

iPhone 6s (A9)

Seco

nds

30FPS-capable

Recent smartphones equipped with ML accelerators can super-resolve gaming streams in realtime.

Energy overhead90

80

70

60

50

40

30

20

10

0

100

Low(iPhone Xs)

Med(iPhone Xs)

Low-SR (iPhone Xs)

Low-SR(iPhone 11 Pro)

%ch

arge

rem

aini

ng

} -5% after 30 min streaming

LevelUp has small energy overheads even with super-resolution enabled.

Conclusion

• Game livestreaming is expensive due to realtime transcoding• LevelUp can greatly reduces game livestreaming costs by leveraging

smartphones for transcoding• LevelUp uses super-resolution to boost quality for reduced-resolution

videos by up to 88%• LevelUp can transcode and super-resolve game streams in realtime

using recent smartphone hardware accelerators

Discussion

• Strengths• Clean and novel approach• Evaluation is extensive and relevant• Tradeoffs are well studied

• Weaknesses• Too much background information• System should be hybrid: cloud can still be helpful• Simple cost analysis could be included

• Mobile bandwidth and resources are costly

levelup: a thin-cloud approach to

Documents