1“Cross-Industry Preservation Architectures” – PASIG May, 2011

Cross-IndustryPreservation Architectures

Michael PetersonMay 2011

2“Cross-Industry Preservation Architectures” – PASIG May, 2011

Introductions

●Michael Peterson Founder, past President, and Chief Strategy

Advocate for the SNIA Currently working on Cloud Archive and Long-term

Retention standards, best practices, market education Information Services Architect – consulting in long-

term retention and digital preservation system design and implementation

Currently driving the LTDPRM.org & ILM2.0.org Communities

Author: “100 Year Archive Requirements Study,” 2008, “Building a Terminology Bridge: Guidelines for Digital Information Retention and Preservation Practices in the Datacenter,” Sept. 2009

3“Cross-Industry Preservation Architectures” – PASIG May, 2011

www.ltdprm.org

4“Cross-Industry Preservation Architectures” – PASIG May, 2011

Agenda

●LTDP Paradoxes (Laws designed to be broken)

●New Digital Preservation Models That the cloud empowers

●Using the Cloud for Digital Archives (Digital Preservation)

5“Cross-Industry Preservation Architectures” – PASIG May, 2011

4 Paradoxes of Digital Preservation

●Data will be lost●Migration does not

scale●Access & use models

keep changing●Cost overwhelms

everything complexity does not

6“Cross-Industry Preservation Architectures” – PASIG May, 2011

How do we break them?“Old” Laws of Digital Preservation

7“Cross-Industry Preservation Architectures” – PASIG May, 2011

100 Year Complexity Barrier

●Overwhelming growth, cost, change Constant Physical and Logical migration Power, cooling, space, people, resources,

maintenance,… Always adding & migrating systems, networking,

storage Managing thousands of formats Constant Auditing and recovery of damaged or lost

data Thousands of moving parts Complex systems and architectures Changing software platforms

8“Cross-Industry Preservation Architectures” – PASIG May, 2011

Aha!

Move from “physical” preservation architectures and design as in physical media or a physical

repository (a 2002 ‘OAIS’)to

“virtualized” Preservation Services based on Service Management principles

(Sounds like the Cloud…)

9“Cross-Industry Preservation Architectures” – PASIG May, 2011

New Digital Preservation Models

Using the Cloud

10“Cross-Industry Preservation Architectures” – PASIG May, 2011

“Physical” Doesn’t Scale

●Old Architecture “Storing digital images

effectively requires standards related to the storage media, such as CD- ROMs, and the file formats, such as TIFF.” Source: “A Resource List for Standards Related to Digital Imaging” Dec. 2010

Physical Application & Storage infrastructure

●Physical Standards Architecture: OAIS 2002 Metadata: Dublin Core:

ISO 15836:2009 Storage Media: ISO

18921:2008, ISO 18925:2008, Digitization: ISO/IEC 10918-1/Cor1:2005, ISO/IEC 10918-3/Amd1:1999

File Formats: ISO 19005-1:2005, Adobe TIFF Specification, V6, 1992

Transfer Protocols: ISO 15740:2008

11“Cross-Industry Preservation Architectures” – PASIG May, 2011

“Infrastructure Virtualization”

●New Architecture Media independent System Architecture

virtualized, self-protecting, cloud based, and self healing

Integrated migration & transformation services

Virtualized historical applications hosted in the cloud in specialized containers running in virtual machines

●New Standards Architecture: OAIS 2010 Metadata: FCIS, PREMIS,

IETF Cloud: SNIA-CDMI Interoperability: NIST

Smart Grid Framework, Cloud and Interoperability workgroups

Object Containerization: SNIA-SIRF

12“Cross-Industry Preservation Architectures” – PASIG May, 2011

Add “Information Virtualization”

●Portable Information Objects Extensible Preservation Objects Location, media independent Secure, auditable, authentic,

portable Self-healing

●On-demand, virtual emulation “Jumpbox” hosted emulators Populations of legacy ‘readers’ Web-based delivery and access

●New Standards Architecture: OAIS 2010 Metadata: FCIS, PREMIS,

IETF Cloud: SNIA-CDMI Interoperability: NIST

Smart Grid Framework, Cloud and Interoperability workgroups

Object Containerization: SNIA-SIRF and CDMI

13“Cross-Industry Preservation Architectures” – PASIG May, 2011

Move to “Managed”

Content Management

Service Management ITIL, ITSM, ILM2.0,

Information Governance

Litigation ‘Ready’Preservation begins

at “Creation”Preservation is a new Datacenter Practice

Operating Practices:ITSM - IT Service Mgmt.

ITIL-IT Infrastructure Library

ILM2.0 - Service mgmt. based approach to information mgmt. and automation

Regulatory Compliance

14“Cross-Industry Preservation Architectures” – PASIG May, 2011

And to “Virtual Services” in the Cloud

●Platform as a Service

● Infrastructure as a Service

●Storage as a Service●Evolving Web access

and use models●Private, Hybrid,

Public Clouds Multiple clouds, multiple

providers

21“Cross-Industry Preservation Architectures” – PASIG May, 2011

Using the “Cloud” in Preservation

●Most likely use-cases: Private and Hybrid

clouds Virtualize infrastructure Virtualize delivery and

access Virtualize emulation Virtualize information

Providing portability

●Examples Web-access models Web-drop boxes Agile, Scalable, cost

effective compute and storage resources (on demand)

Virtual emulation Demand spikes

Disaster recovery Distributed data sets Infrastructure extensions

22“Cross-Industry Preservation Architectures” – PASIG May, 2011

Emerging Cloud Standards

●Cloud Data Management Interface, CDMI SNIA to ISO: storage-to-cloud, cloud-to-cloud

interchange format

●Self-contained Information Retention Format, SIRF SNIA to ISO: extensible preservation object format

●Interoperability ISO project: Data Preservation Interchange

Framework, DPIF

23“Cross-Industry Preservation Architectures” – PASIG May, 2011

Cloud Data Management Interface

●Data Portability Standard with an Object Storage Interface Move data and metadata in standard portable

containers in and out of the cloud and between clouds

Simple XML container of objects plus metadata

●A data and information services management interface and control path Operate services through CDMI

Rules and Policies in metadata Cloud Peering – cloud to cloud communications

24“Cross-Industry Preservation Architectures” – PASIG May, 2011

Design for the Cloud

●Considerations Establish Service

Objectives Include verification of

recovery, authenticity, availability, digital audit, etc.

Consider using multiple cloud destinations or local and remote copies for increased reliability and availability

Beware of excessive moving of data across the WAN due to high I/O and bandwidth costs

●Evaluate Cloud providers Establish strong contracts

●Test and Audit All required services

●Use CDMI !

25“Cross-Industry Preservation Architectures” – PASIG May, 2011

Cloud Contract Considerations

CostsRetention

Management Preservation/Integrity/

Authentication Return and Secure

Disposal – Subpoenas, Control

Legal Hold Digital Audits &

Verification Physical and logical

migration practices and authenticity verifications

•Access Availability, Protection,

Security & Confidentiality

Search/Discovery Multi‐Cloud Provider

Relationships

• Right to Conduct Forensic Exams

•Cross‐Border Data Transfers

26“Cross-Industry Preservation Architectures” – PASIG May, 2011

Summary ThoughtsPreservation Architectures: Virtualization and Cloud

27“Cross-Industry Preservation Architectures” – PASIG May, 2011

Move to Virtual Preservation

●Shift thinking from “Physical” Preservation to “Virtual”

●Virtualization Applies in many ways System, storage, application, infrastructure Information Migration – both physical and logical Cost reduction

●Conclusion: ‘Cloud’ has a positive role

28“Cross-Industry Preservation Architectures” – PASIG May, 2011

Using the Cloud

●Start out Private, Move to Hybrid●Apply Service Management Principles

Classify, Requirements, SLAs, Design, Audit, Improve

●Design for the Cloud Create strong and measureable SLA style contracts Test, Audit, Verify

●Use and Promote CDMI Need cloud interface, management, and

information portability standards

29“Cross-Industry Preservation Architectures” – PASIG May, 2011

Contact Information

●Michael Peterson IMERGE consulting and LTDPRM.org mpeterson@ltdprm.org (805)201-3178