chesa-aws preservation presentation may 2019...• implementing data archival best practices to...
TRANSCRIPT
CHESAPEAKE SYSTEMSMEDIA TECHNOLOGY ARCHITECTS AND WORKFLOW SOLUTIONS PROVIDER
• Scaling Technologies and Media Workflows to support growth.
• Creative content is not being produced at the pace and quality that the business needs.
• Implementing Disaster Recovery and Business Continuity strategies.
• Enhancing internal support offerings for technologies and workflows.
• Implementing systems and workflows to support monetization of existing content.
• Implementing data archival best practices to preserve and protect existing content.
• Confidence is lost in current support resources; the current provider lacks the appropriate resources, coverage, and experience to handle a crisis.
Media Content Creators turn to Chesapeake Systems when:
CASE STUDY: STUDIO ARCHIVEScaling ingest to support growing content library
ARCHIVE MISSION• Provide Archive as a service
to the organization
• Store & Protect Studio history
• Provide library search & retrieval services
• Enhance search with technical and qualitative metadata
• Provide processing services
Sidecar and Metadata
Captions, Metadata, Subtitles, Manifests, LUT Tables, etc
Packages
Image Sequences, Games, iTunes Packages, etc
Application Project Files
Documents Media Files
PRODUCTS IN THE ARCHIVE
ON-PREMISES WORKFLOW
Vendors submit data packages into the archive over WAN
Data packages arrive on cache storage, triggering
MAM workflow to validate and process
Proxies are created for customers to browse in
the MAM and trigger restores
Checksums are processed for
validation of ingest and integrity checks
Upon successful tape archival an API call
updates Front End MAM with success and
checksum validation
Upon successful tape archival an API call
updates Front End MAM with success and
checksum validation
Site #2 can serve as an “air gapped”
site, duplicate tapes vaulted
Restored data flows back directly from archive
site(s) to the Studio, with automation triggered from
the Front End MAM
WORKFLOW CHALLENGES
Vendor submissions increase in quantity and
package size consistently
New and changing video codecs require frequent
package updates to transcoders for
compatibility
Hardware requires 3-9 months budgeting and
procurement process to expand and support
ingest growth
Network bandwidth across all paths needs to increase
with ingest growth
A disruption to tape production will halt
ingests
All Systems require ~5 year
refresh
Tape technology refresh may require full restore
and re-archive to be automated and
monitoredDevOps and Admins have
roadblocks to managing and developing against the sites
HYBRID APPROACH
Vendors submit data packages into the
archive over WAN or Direct Connect
Expansion of Ingest capabilities can be
handled on demand with Spot Instances
First Archive data set stays in the cloud,
limiting total egress
Restores triggered in MAM are sourced from on-premises archive to
limit total egress
Transcode as a service eliminates managing
updates for format compatibility
AI/ML as a service can be leveraged on demand from cloud archive to
enhance metadata
MAM Architecture is scalable and
replicated through containerization and
serverless architecture
On-premises tape archive keeps offline “air gap” by vaulting
duplicate tapes
Two separate archive storage technologies
protect the archive from external threats
caused by markets
DevOps and Admins now have minimal roadblocks to developing against the
environment
MIGRATION TO HYBRID
Periodic data integrity checks require data retrieval
Piggyback off of integrity checks and
send data to S3Migrate database
records from retained archive site
Lambda can monitor S3 bucket for migration
data. Triggers MAM to run workflow to validate
files and link data objects to placeholder
records from DB migration.
MAM workflow can perform API calls to on-
premises system to trigger purge of local
copy from disk cache, no need to egress data
back to physical archive
Migration is limited by both network speed
and disk/tape speed at source.
QUALITIES OF THE MAM• Taxonomy System (or ability to integrate with one)
• Customizable Objects - no organization is equal
• Hierarchical object and asset structures to support “complex assets” and packages
• APIs and Metadata Exchange capabilities
• Workflow Automation/Business Process Management - with clear visibility into processes
• Agile Workflow Development capabilities
• UIs catering to multiple sets of required User Experiences
• Modernized (cloud ready) software architecture - easily scale and manage costs
• Dashboards/Analytics - track processes, library statistics, and costs