Choosing the Right IP Cores for Low-Latency Video Streaming

Nikos Zervas CEO CAST, Inc. December 2016 www.cast-inc.com

Slide2LowLatencyVideoStreamingIP

Introduction

  Low latency is important when there is real-time interaction with the video content   Gaming, Video Conferencing, Automotive, Industrial, Medical,

Defense

  How low is low latency?   For human interaction sub-100ms is very low latency   Machine interaction may require even less

Slide3LowLatencyVideoStreamingIP

Introduction

  End-to-end or glass-to-glass latency is the amount of time between the instant a frame is captured to the instant the same frame is displayed

  Latency is introduced by temporal storage (buffering)   Latency is measured in time or buffering (storage) units

Slide4LowLatencyVideoStreamingIP

Latency comes from buffers Mission: Eliminate Excessive Buffering

  Buffers are hidden within IP cores, and IP cores may impose external buffers

Focus on what really counts:   Pixel buffers do not really count   2000 pixels: 0.03 msec for 1080p30

  Video Line buffers count   100 lines: 3.01 msec for 1080p30

  Frame buffers count even more   1 frame: 33.33 msec for 30fps

Oneframe 33.33 msOneline 30.86 μsOnepixel 16.08 ns

LatencyFor1080p30Video

Slide5LowLatencyVideoStreamingIP

Choosing the Right IP Core

Throughput is irrelevant: Two cores with same throughput may have totally different latency

Input/output data-ordering may impose external buffers for reordering

Choose “line-based” vs “frame-cased” image processing algorithms

Some cores do not produce any output before they process a full frame, and some do not start processing unless a full frame is available at their input.

TIP

Slide6LowLatencyVideoStreamingIP

Glass-to-Glass Latency

Why do I even need a BRAB and a DSB?

ProcessingStage Buffering Latency(for1080p30)

Capturepost-processing(e.g.bayerfilter,3A,ISP)

Fewlines(e.g.32) <1msec

VideoCompression(JPEG,MPEG-1/2/4orH.264)

8or16linesforconversionfromrasterscanFewthousandofpixelsontheencoderpipeline

0.5msec<0.05msec

Bit-RateAveragingBuffer Anumberofframes(e.g.morethan30)tosub-frame(e.g.1/2frame)

from1secto16msec

NetworkProcessing(e.g.RTP/UDP/IPencapsulation)

FewKbytes <1msec

DecoderStreamBuffer Anumberofframes(e.g.morethan30)tosub-frame(e.g.1/2frame)

from1secto16msec

VideoDecompression(JPEG,MPEG-1/2/4orH.264)

8or16linesforconversionfromrasterscanFewthousandofpixelsonthedecoderpipeline

0.5msec<0.05msec

Displaypre-processing(e.g.Scaling,ChromaResampling)

Fewlines(e.g.32) <1msec

Slide7LowLatencyVideoStreamingIP

Bit-Rate Control and Latency   Bit rate is not natively constant nor predictable

  Bit rate control averages the bit rate to a constant value

0

5

10

15

20

25

0 100 200 300 400 500 600 700 800 900 1000

Mbp

s&me(msec)

0

5

10

15

20

25

0 100 200 300 400 500 600 700 800 900 1000

Mbp

s

&me(msec)

1stAveragingPeriod 2ndAveragingPeriod 3rdAveragingPeriod

Slide8LowLatencyVideoStreamingIP

BRAB & DSB

!!

!!

  Bit-Rate Averaging Buffer (BRAB) on the encoder side “normalizes” encoder bit rate   Can be reduced to a zero-latency FIFO, if channel can “absorb”

bit rate picks

  Decoder Stream Buffer (DSB) buffers one averaging period worth of stream, before decoding is initiated, to avoid video playback interruption

BRAB

Slide9LowLatencyVideoStreamingIP

BRAB & DSB

!!

!!ü BRAB Latency = DSB Latency = Averaging Period

BRAB

0

5

10

15

20

25

0 100 200 300 400 500 600 700 800 900 1000

Mbp

s

&me(msec)

1stAveragingPeriod 2ndAveragingPeriod 3rdAveragingPeriod

Slide10LowLatencyVideoStreamingIP

Rate Control is What Matters

  All rate control algorithms adjust the compression-level based on the remaining bit rate budget in the averaging period (=buffer occupancy)

ü  Accurate prediction of future content’s compressed size is infeasible, but some algorithms make better predictions by:   Assessing the “complexity” of the remaining content   Getting “trained” based on history

ü  The frequency of compression-level adjustments varies §  More frequent adjustment allows to quickly adopt to content

changes and correct inaccuracies of predictions

Slide11LowLatencyVideoStreamingIP

The Role of Rate Control

Rate Control defines Latency but also Quality

  Rate Control is not described in standards

Samecompressionalgorithm,bitrateandaveragingperiod–DifferentEncoders

Slide12LowLatencyVideoStreamingIP

Choosing the Right IP Core

Focus on encoder’s rate control attributes, and evaluate latency and quality (IP core vendor should be able to help you) Test streams do not always stress rate control. Rate control algorithms may overflow or underflow BRAB & DSB, unless they are able to regulate compression levels multiple times within the averaging period

Regardless of rate control attributes, software codecs are not suitable for ultra-low latency

TIP

Slide13LowLatencyVideoStreamingIP

Choosing the Right H.264 IP Core

Choose an H.264 encoder IP core supporting Intra-Refresh

Intra-Refresh allows construction of an intra-coded reference over a number of frames. It eliminates I-frames and the related bit-rate “peaks”

TIP

0

5

10

15

20

25

30

35

0 200 400 600 800 1000

Mbp

s

&me(msec)

0

5

10

15

20

25

30

35

0 200 400 600 800 1000

Mbp

s

&me(msec)

IPPcoding Intra-Refreshcoding

Slide14LowLatencyVideoStreamingIP

JPEG

Choosing the Right JPEG IP Core

A JPEG encoder IP core is not suitable for sub-frame latencies, because the JPEG standard only allows Quantization changes at frame boundaries

Some JPEG encoders can regulate bit-rate also by truncating code-bits from Huffman-coded blocks. The stream remains complaint, and allows intra-frame bit-rate “corrections”

Slide15LowLatencyVideoStreamingIP

Compression Cores from CAST

PLUS:ü  HardwareRTP,UDP/IPstacksü  CompleteVideo-Over-IPplaTormsü  ExperVsetohelpyouarchitectyoursystemandselecttheright

algorithms

CompressionStandard SuggestedMinAveragingPeriod

CompressionLevels

MotionJPEG 1/4Frame ModerateIntra-onlyH.264 1/4Frame HighH.264wIntraRefresh 1/4Frame VeryHighH.264w/oIntraRefresh 3-4Framse VeryHigh