avid's stereoscopic editorial...

Shailendra Mathur, Video Chief Architect | March 19th 2013

Avid's Stereoscopic Editorial Architecture

Light Fields, Intelligent Computing and Beyond

2 Confidential and Proprietary Information. Please do not copy, forward, redistribute, or publish. Avid ©2013.

• Stereo 3D data handling

• Avid Intelligent Compute

Architecture

Topics


Avid introduction

Tools for Film, Broadcast and Live productions

Video and Audio

Focus today on Film editorial editing tool – Media Composer

Our customers are Academy, Grammies and Emmy award winners, • … and so are we!

4

Stereo 3D Data Handling


Trends

Increasing granularity (resolution) of video

High frame rate usage

High spatial resolution

High bit-depth

Increasing number of views

Stereo shooting - Mono and S3D deliverables

Plenoptic/Light Field Cameras

Camera Grids


Our Vision!

“Super-sample the work, edit it later!”

The Story-teller editing the film wins!

More elements of the scene to work with • Zoom-in,

• Pan and scan

• Slow-down,

• Speed-up

• Increase contrast,

• Expose new dynamic ranges

…without losing display precision!


Problems to solve

Need a data model for managing multiple views and resolutions

Support common editing and data management functions and workflows


Light Fields – a parametric model

• 7-D Plenoptic Function

• Camera streams are

spatio-temporal-

colorimetric view-

samples of

“Radiance received along

any direction V arriving at

any point E in space, at

any time „t‟ and over any

range of wavelength”

),,,,,,( tEzEyExvvPI


Multi-view sampling of a light field

Spatial

Color

Temporal

http://commons.wikimedia.org/wiki/Main_Page

Light Field


Multi-cam productions

Temporally aligned multi-cam group

Time Reference

C1 C2

C4

C3

Sync


Registered multi-cams

C1 C4 C3 C2

Temporal synchronization

Spatial synchronization

Color synchonization

Scene


Multi-channel video data-model

C2 C1

C-Comb

Cn

S3 Sn

Channel

Grouper/Synchronizer/Resampler

Channels

Streams

Sm

…

…

C2 C1

C-Comb

Cn

S1 Sn S1

…

… S1 S2

Resolutions

Editing

system

Scene 1 Scene 2

…

Scene Z

C-Comb

C1

Light Field

13 Confidential and Proprietary Information. Please do not copy, forward, redistribute, or publish. Avid ©2013. Resolution 3

S4 L DNX444

Resolution 2 Resolution 1

Proxy Editing and Multi-Mastering

C2:

Right Eye

C1:

Left Eye

(Leading)

Channel

Combiner

Output context:

SidebySide, L|R,

DNX36,

L|R audio

S1 L 1:1 S2 R 1:1 S3 R|L 1:1

Output context:

Full Frame, L|R,

uncompressed,

L|R audio

Output context:

Full Frame, Leading Eye,

DNX444,

L|R audio

S5 with

Right Audio

Stage3 Stage1

Stage2

S4 with

Left audio

C4:

Right Audio

Capture full frame

uncompressed S3D

left and right views

Transcode to side by

side S3D

Full Frame S3D

Master output

Mono Master output

using leading eye

C3:

Left Audio


• Method of grouping multiple views

• Spatial, temporal and Color synchronization of views

• Method of arranging resolutions under the views

• Resampling of scene based on output context.

Data Model – Problem solved.

15

Avid Intelligent Compute Architecture


Performance Issues

Multiple views => more processing

• codecs,

• visual effects

• graphics,

• visualization

Higher resolution, frame rates and bit-depth


Avid Intelligent Compute Architecture

Heterogeneous compute

Hardware

• Intel multi-core CPUs (HP

Z800/820, Apple Mac Pro )

• Nitris DX w/ dual DNxHD or AVC-I

codec acceleration modules

• NVIDIA GPU

• NVIDIA 3DVision support

Intelligent Media Player

http://www.google.ca/url?sa=i&rct=j&q=intel%C2%AE+xeon%C2%AE+processor+x5690&source=images&cd=&cad=rja&docid=NwjV129BAp2OAM&tbnid=uzyaUc0z3E8xEM:&ved=0CAUQjRw&url=http://www.ebay.com/itm/FREE-SHIP-Intel-Xeon-Six-Core-Processor-X5690-3-46GHz-6-4GT-s-1366-pin-12MB-CPU-/200867241483?pt=CPUs&hash=item2ec49ede0b&ei=T3RCUYjiBeWbygHFooGICw&bvm=bv.43287494,d.aWc&psig=AFQjCNFttlHtskNsProLz1kK8AU8_9gkYA&ust=1363396010391400


Multiple pieces of code for each compute domain

Balancing across multiple compute domains dynamically

Optimal transfers between hardware?

Dynamically adding a new hardware processing domains

Problem with heterogeneous compute

CPU

Decoder Encoder Resize Keying

Application Code

FPGA GPU Low-level run-times/drivers Compute X

Storage Storage Data



How others have solved it

Main philosophy - "Write

once, support many"

• Generic Language with HW

specific compilation, e.g. OCL

• Code generation for specific

compute models, e.g. Cg

CPU

Decoder Encoder FX1 FX2

Application Code

FPGA GPU Compute X

Storage Storage

Low-level run-times/drivers

Generic Language/Macros

Data



How we solved it

Philosophy - "Optimize

Many, Host once"

• write compute domain

specific optimized code

• single API contract with

client application.

• Domain-abstracted player CPU


Application Code

FPGA GPU Compute X

Storage Storage


Shell Processor Components

Compute-Optimized processor Components

Data

ACPL

CPU


Application Code

FPGA GPU Compute X

Storage Storage


Shell

Procs Decoder

Proc

Encoder

Proc FXProc1 FXProc2

Compute-

optimized

Procs

Decoder

Proc - CPU

FXProc1

CPU

Decoder

Proc - FPGA FXProc1

GPU

FXProc1

CPU FXProc1

Compute X

Encoder

Proc - CPU

Encoder

Proc

Compute X

ACPL

Data




Plug’N Play Architecture

Home grown component model

• Similar to COM

Plug and play

• Compute domains

• Processors

• Data types and formats

• Converters


Plugging-in NVIDIA GPU - Benchmark results

Task How much faster with NVIDIA GPU?

4 streams with title 2.6x

4 color corrections, 3 picture-in-picture with soft border, 1 title

4 streams with title 7.3x

4 color corrections, 3 perspective (3D Warp) with soft border, 1 title

2 advanced effect streams 1.8x

1 color correction + stabiliser + spectramatte key + resize + Symphony advanced color correction

1 FullFrame stereo effect stream 1.4x

color alignment + positional alignment + S3D rotational alignment + vergence + zoom + floating window

2 FullFrame stereo effect streams 1.9 x

2 color alignments + 2 positional alignments + 2 S3D rotational alignments + Depth transition + Dissolve

System • Avid Symphony 6.0

• HP Z800 Workstation

• Windows 7 64-bit

• 6 GB RAM

• NVIDIA Quadro 4000

• Avid Nitris DX

• Avid VideoRaid storage

Material • AVC-Intra 100 sources rendered

to Avid DNxHD 185x

• 1080i/50 10-bit

• Processing:

• Full resolution HD

• Full-Frame processing of Stereo 3D streams 16 bit processing precision

• Render quality: best (advanced polyphase)


Example AVC-I Decode + color alignments + 2 positional alignments + 2 S3D rotational alignments + Depth transition + Dissolve + DN220x encode

Stereo Full Frame AVC-I decode

color alignments per stereo pair

positional alignments per stereo pair

S3D rotational alignments per stereo Pair

Depth transition

Dissolve DNx220x

10bit Encode

Inter-Domain

Data Converter

S1L

Decoder

S1L

Color

Correct

Storage Storage

S1R

Decoder

S2L

Decoder

S2R

Decoder

S1R

Color

Correct

S2L

Color

Correct

S2R

Color

Correct

S1L

PanZoom

S1R

PanZoom

S2L

PanZoom

S2R

PanZoom

S1L

Perspective

Rot

S1R

Perspective

Rot

S2L

Perspective

Rot

S2R

Perspective

Rot

S1L

HIT

S1R

HIT

S2L

HIT

S2R

HIT

Dissolve

Left

Dissolve

Right

Encode

Left

Encode

Right

Inter-Domain

Data Converter


avid's stereoscopic editorial...

Documents