denis leconte€¦ · calypso update over the past few months:-vfx pipeline-tape-based...

Denis Leconte

Calypso UpdateCalypso Update

Over the past few months:

- VFX Pipeline

- Tape-based pipeline

- LTO5 migration

- Infrastructure upgrades

5/5/2011Production Backbone / Calypso Update

OverviewOverview

The Smurfs Movie

- First project to use Calypso

- Started mid-2010

- VFX heavy

- 2D to 3D Conversion

- Currently in DI

“The 2D Smurfs conform is going amazingly well. “

Jay Warren, Project Manager


The ChallengeThe Challenge

Multi-vendor VFX Delivery

- Needed to centralize deliveries

- Needed to validate them

- Delivery specs are not always respected

- Ability to fix simple problems

- Delivering “clean” material to Editorial

The Smurfs Delivery Spec

- Name shots starting at 1001 on FFOA- If possible, at a slate at the head (frame 1000)- Header timecode should start at 01:00:00:00 on slate (01:00:00:01 on FFOA)- Directory structure shot/resolution/shot.number.dpx- Follow the shot naming conventions as requested by production

045CD_050_x00001

% of vendors delivering shots just like this?


Solution: Replacing this:Solution: Replacing this:

DPX Files QT or MXFs

EDL

2 Separate paths – possibility for 2 Separate paths – possibility for disconnectdisconnect


……With this:With this:

DPX Files

EDL

QT or MXFs

Single Path: Single Path: No disconnect!No disconnect!


VFX Dailies: Validation toolsVFX Dailies: Validation tools

Run on every incoming VFX shot

Performs a technical check

Validation pipeline

- Sequence checker: are frames in a continuous sequence (otherwise: reject shot)- File checker: are files legitimate DPX/TIFF/EXR files (otherwise: reject shot)- Naming convention verification: does the directory structure and shot naming follow conventions?- Number convention validation- Header timecode validation


VFX Dailies: Correction toolsVFX Dailies: Correction tools

Run on shots that have correctable noncompliance errors

Correction pipeline

- Directory reorganization- Header timecode rewriting- Renumbering/Padding change


VFX Dailies: Putting it togetherVFX Dailies: Putting it together

Essential requirements

- Unobtrusive

- Automated as much as possible

- Try and reject as few shots as possible (fix as much as one can)

- Detailed notifications to Production

Automated system

- Frequent schedule runs- Propagates data from Isilon to BB Staging directory- Validates- Notifies of rejects- Moves data to BB Asset storage area- Fixes what can be fixed- Registers assets- Generates Quicktimes/ALEs/MXFs- Pushes Editorial material back to Isilon- Notifies all concerned parties of new assets- Separate emails to BB admin for exceptions


Data Flow and Isilon IntegrationData Flow and Isilon Integration


In Progress: VFX InspectorIn Progress: VFX Inspector


Recent DevelopmentsRecent Developments

Asset Request Processing

- On a SAN

- On a tape robot

- Potentially more than one of each

- All tracked in Calypso

- Asset selection is made for shortest time to delivery

Retrieve it


Asset request processingAsset request processing

The assets for a request can be in a number of places:

- On a SAN

- On a tape robot

- Potentially more than one of each

- All tracked in Calypso

- Asset selection is made for shortest time to delivery

Retrieve it


Cache managementCache management

System to keep track of:

- Unique request count per asset

- Asset age in cache

Automated cache expiration, based on target cache size, targeting least used/older assets.


LTO migrationLTO migration

- All LTO4 content migrated to LTO5.

-1,775 tapes

- Done with no impact on production archival/retrieval

- Scheduling system written to optimize robot use

- Efficient use of operator time (only 45 hours for the complete migration)

Tape operations priorities

(Implemented using Rush and TSM)


Additional Tape Archive Additional Tape Archive DevelopmentsDevelopmentsTape scheduling expert system

- Retrievals often involved several operations on the same tape

- System tries to minimize tape contention

- Retrieval sequencing- Using multiple sets

Throughput improvement

- From 50MB/s (per drive) to over 100MB/s

Compression Staging

-To compress/decompress media backups on the fly

LTFS

-Set up LTFS archive/retrieve capability

-Used for 21 Jump Street


Asset “Heat Map”Asset “Heat Map”

Asset activity is tracked

- What assets were pulled

- How many pulls

Can prepare a list of everything that was used

- And everything that was not

- We are ready for EDL-based archival (of only what was used)


REDRED

RED files ingested natively

- Asset metadata extracted and part of Calypso registration

Media pulls

- R3D, DPX, RMD, Metadata

- Hybrid REDline + Nuke workflow for best image quality


ARRIRAWARRIRAW

Currently: Metadata only

- Working on licensing of ARRIRAW format and SDK

- Header documented in-house

- Asset registration with metadata

More to come…


CODEX CompressedCODEX Compressed

Codex can record compressed

- Native handling of compressed files

- Metadata aware

- Ability to transcode on the fly to DPX files

- Bandwidth/Storage savings

- 3:1 compression tested and satisfactory


Improved Editorial/Conform Improved Editorial/Conform FeaturesFeatures- Interactive feedback on EDL request

- On-the-fly EDL correction- EDL modification/rewrite

- More sophisticated “Shopping Cart”

- Persistence- Customizable processing

parameters


More Camera/Workflow SDKsMore Camera/Workflow SDKs

-Plug-in architecture

-Make it as simple as possible to add a new system


-Sony F65

-Sony F3 (MPEG2 within MPEG4 wrapper)

-3D F35/F3 workflow

Dailies Validation/AssistDailies Validation/Assist

- System for thorough dailies/footage metadata review

- Naming conventions- Timecode- Directory organization

- Necessary for Backbone ingest- Naming conventions- Timecode- Directory organization


- Plug-in architecture here too- Dailies organization tend to be

very show-specific- Some common elements,

many differences

CompressionCompression


- Aiming to make compression a near-transparent process

- Implemented as a standard “data move” plug-in

- Also available to/from tape

- JPEG2000-based- In the process of licensing

Kakadu- Additional enhancements

added for metadata handling

- Do be deployed mid-may 2011- Study of optimal compression

rates/options under way

Web Services Web Services

-SOAPify targeted Calypso services

- Query for new assets- Query for update assets- Query for updated asset

metadata- Asset updates- Metadata update- Asset deletion - Asset export- Title search


-MBC/EAGL integration- Web service based- Also including processing

services

Denis Leconte

denis leconte€¦ · calypso update over the past few months:-vfx pipeline-tape-based...

Documents