information systems and closure in archival keys to the digital … grow ² increasing volumes of...

Open CloseGrow

Evolve

Openness, Growth, Evolution,Openness, Growth, Evolution, and Closure in Archivaland Closure in Archival Information SystemsInformation Systems

Lessons from NARA’s Experience

September 2008

Kenneth Thibodeau, Director Electronic Records Archives Program

National Archives and Records Administration IEEE Symposium on Mass Storage Systems & Technologies

Keys to the Digital Future

Open CloseGrow

Evolve Archival Information System

Conceptually: “an archive, consisting of anorganization of people and systems, thathas accepted the responsibility to preserveinformation and make it available for a Designated Community.”

• ISO Reference Model for an Open ArchivalInformation System (OAIS). ISO 14721:2003

Empirically: the National Archives’ OpenArchival Information System, theElectronic Records Archives

Open CloseGrow

Evolve

What is the Electronic Records Archives (ERA)?

ERA is the system the National Archives and Records Administration (NARA) is developing to

Reengineer and automate the lifecycle management of all types of records of the U.S. Government Preserve and provide sustained access to electronic records of the U.S. Government

Open CloseGrow

Evolve ERA Development Timeline

Full OperatingFull Operating CapabilityCapability

ERA Base System

ERA Search & Access System

Enhancement

Initial Operating Capability (IOC)

6/08

Operation & Maintenance

9/05 9/06 9/07 9/08 9/09 9/10 9/11

Enhancement

Enhancement

Enhancement

Open CloseGrow

Evolve

ERA Base System Development

Focus: Federal Records National Archives

IOC Functions (2008): Creation, review and approval of records schedules Requests to transfer records, transfer of physical and legal custody Transfer, inspection, and archival storage of electronic records

Open CloseGrow

Evolve Initial Users

Open CloseGrow

Evolve

ERA Search and Access System Development

Initial Focus: Electronic records of the Executive Office of the President, G W. Bush Presidential Libraries

100 TB Functions:

Rapid ingest & indexing • Transformation to more accessible form.

Archival storage Full content search Basic case management for special requests

Open CloseGrow

Evolve Future DevelopmentFuture Development

Public Access to Any information about records

• Ordering of copies of records Electronic records stored in the system

Long-term preservation of electronic records Ability to use a variety of techniques simultaneouslyand over time

Review and redaction of sensitive content Support for Federal Records Centers Exponential growth in stored data

Keys to the DigitalKeys to the Digital FutureFuture

OpennessOpennessGrowthGrowthEvolutionEvolutionClosureClosure

Lessons from the ERA experience

Open CloseGrow

Evolve

Open CloseGrow

Evolve OpennessOpennessAn Archival Information System needs to be open to

New types of electronic records

Rising and changing user expectations

Creative approaches to meeting the challenges of electronic records and demanding users.

Open New Types of Records: Geographic Information Systems

OpenNew Types of Records:

Product Data Computer Assisted Computer Assisted Design Engineering

Product Computer Analysis and Testing Assisted Manufacture

Open New Types of Records: Critical Infrastructure Data

Source: CLindberg http://commons.wikimedia.org/wiki/Image:I35_Bridge_Collapse_4crop.jpg

Open New Types of Records: Virtual Reality: Crime Scene Investigation

Open New Types of Records: Medical Tests and Observations

Open CloseGrow





Rising and Changing User Expectations

Open

Open CloseGrow





Open Creative Approaches

The conceptual apparatus we bring to bear on – The nature of records – Requirements for preserving records – Requirements for serving users

Open Creative approaches: Partnerships NationalScienceFoundation

NIST San Diego

SupercomputerCenter

National Computational Science Alliance Global

Grid Forum

Army Research Laboratory


Openness GrowthGrowthEvolution Closure

Open CloseGrow

Evolve

Open CloseGrow

Evolve GrowthGrowthAn Archival Information System needs to be able to grow to

Process, store and provide access to increasing volumes of electronic records

Accommodate increasing numbers of users and frequency of use

GrowIncreasing Volumes

of Digital Information • In 2006, the amount of digital information created,

captured, and replicated was …281 exabytes or 281 billion gigabytes. This is about 3 million times theinformation in all the books ever written.

• By 2011, the digital universe will be 10 times the size it was in 2006.

• Not all information created and transmitted gets stored, but by 2011, almost half of the digital universe will nothave a permanent home.

• The number of electronic information “containers” — files, images, packets, tag contents — is growing 50% faster than the number of gigabytes. The informationcreated in 2011 will be contained in more than 20 quadrillion — 20 million billion — of such containers

– IDC. The Diverse and Exploding Digital Universe. An Updated Forecast of Worldwide Information Growth Through 2011. March 2008

Grow Transfers of Digital Files to NARA 16,000

14,000

12,000

10,000

8,000

6,000

4,000

2,000

01970-1988 1989-1995

Grow Transfers of Digital Files to NARA 250,000

200,000

150,000

100,000

50,000

01970-1988 1989-1995 Reagan/ Bush

Grow Transfers of Digital Files to NARA

0

5,000,000

10,000,000

15,000,000

20,000,000

25,000,000

30,000,000

35,000,000

1970-1988 1989-1995 Reagan/ Bush Clinton

Grow Planning for Open-ended Growth

DREN

Internet

NARANET

Public

SAS ERA

Future ERA Instance

System Management

OtherAgencies

Instance Instance Base ERA

Federation


Openness Growth EvolutionEvolutionClosure

Open CloseGrow

Evolve

Open CloseGrow

Evolve EvolutionEvolutionAn Archival Information System needs to be able to evolve in response to

Changing Information Technology • Obsolescence

• Opportunities

Changing business requirements

Evolve Obsolescence of Formats of Electronic Records

• Strategy: Preservation and Access Levels – Common:

• Retain records in original formats

– Basic Level: • Use original or contemporary software for access

– Enhanced Level • Create new version in current format, or • Use new software for access to original format

– Ideal Level • Create version in persistent format, or • Create persistent software for management and access

Evolve Obsolescence of Formats of Electronic Records

• ERA System Architecture: – Does not prescribe specific preservation solutions – Allows a variety of different software tools to be

introduced and used for different formats. – Enforces archival requirements

Preservation Framework

Record: Correspondence

E-mail

Attached Photograph

Proprietary to standard image format converter

Standard form Photograph

E-mail to structured data

converter

E-mail Header

E-mail Message

Example

Open CloseGrow

Evolve EvolutionEvolutionAn Archival Information System needs to be able to evolve in response to

Changing Information Technology • Obsolescence

• Opportunities

Changing business requirements

Evolve Preservation Options Sp

ecifi

cG

ener

al

Maintainoriginal

technology

ObjectInterchange

Format

PersistentObject

VersionMigration

FormatStandardization

Typed Object Conversion

Rosetta Stone Translation

Viewer

Re-engineerSoftware

UniversalVirtual

Computer

ProgrammableChips

Emulation

StaticTranslator

DynamicTranslator

VirtualMachine

Multi-Valent

Document

Preserve Method Preserve Technology Objects

Evolve Changing Information Technology: Service Oriented Architecture

Archive Storage VLAN

System/Business Application VLANExternal AccessGateway (DMZ Access)

ERA Management VLAN

Ingest VLAN

WebServer VLAN

IngestWorkingStorage

ArchiveDisk

Storage(Managed Storage)

To GreenbeltERA Mgt

Firewall+ IDS

HW LoadBalancer

Ingest Server(App Server)

Unix Input Workstation

Physical IngestEthernet Switch

WindowsInput

Workstation

Printer

DVD

TapeDrive

CDROM

Printer

DVD

TapeDrive

CDROM

HW Load Balancer

Web Server

Ethernet Switch

Archive LabelPrinting WS

Printer

ArchiveTape

Storage

Mgt Router(w/IPSec)

AccountabilityServer

AccountabilityWS

Backup RestoreServer

EnterpriseSystem Mgt

Server

Help DeskServer

Network Mgt Server

Security Server(Code Modification)

Security Server(Vulnerability Scan)

SoftwareDeployment

Server

Central DataManagerServer

Data ServicesServer (Mgmt

DB Server)

ERAExtranet

GOVTAgencies

NARA

Core Switch(w/FW + IDS + LB)

Firewall + IDS

Firewall + IDS

Firewall + IDS

Firewall + IDS

Ethernet Switch DatabaseServer (T&I)

SOCSOC WSEthernet

Switch

Firewall+ IDS

Ethernet Switch

Firewall+ IDS

DocumentumServer

StrongAuth

Server

DNS/Directory

Server

DatabaseServer(SBA)

AssetMgnt WS

AppServer(SBA)

InstanceDatastore

FCSwitch

Mgmt LAN, Storage Network

Backup Network

Ops LAN,Management LAN, Storage Network

Ops LAN, Management LAN, Storage Network

Ops LAN, Management LAN, Storage Network

Ops LAN, Management LAN

Extranet Router(w/FW + IDS + IPSec)

Firewall + IDS

Firewall + IDS

Firewall + IDS

Firewall + IDS

Derived from: ERA Hardware Block Diagram -2007 0823 (Tab: I1R2 U/USBU Detailed Block) Updated 24 Aug 2007

1Gb Ethernet 2/4Gb Fibre Channel

Evolve Service Oriented Architecture As Built

EOP ERA

System/Business ApplicationEnclave

DMZ Enclave Management Enclave

IngestWorkingStorage

InstanceData store

To GreenbeltWAN

HW LoadBalancer

DVD

TapeLibrary

CDROM

HW LoadBalancer

FC Switches

Mgt Router(w/IPSec)

Firewall+ IDS

Log LogicApplicance

Storage MgtServer

Backup RestoreMaster Server

EnterpriseSystem Mgt

Server

Help Desk Server

Network MgtServer

Security Server(Code

Modification)

Security Server(Vulnerability

Scan)

SoftwareDeployment

Server

Extranet Router(w/FW + IDS + IPSec)

Firewall + IDS

Firewall + IDS

Firewall + IDS

Firewall + IDS

HCAPStorageNodes

Mgmt DB Server

ERAExtranet

GOVTAgencies

NARA

Web/Application

Server

Base ERA WebServer

EthernetSwitch 1

DatabaseServers

HCAPSearchNodes

PrimaryArchiveStorage

Base ERAIngest DB

Server

EthernetSwitches

Backup/ RestoreMedia Server

ECMServer

Tape Library

Ingest Workstation Enclave (Base ERA)

Firewall + IDS

PhysicalIngest

Workstations

PhysicalIngest

Printers

SOC Enclave

Operational ServicesEnclave

DNSServer

ActiveDirectory

Server

LDAP andRadiusServer

StrongAuthentication

Server

SOCWorkstation

Firewall + IDS

USP/V

Base ERA

ManagedStorage

LabelPrinter

Central DataManagement

Servers

EthernetSwitch 2

EthernetSwitch 1

Base ERAIngestServer

EOP IngestWorkstations

EOP IngestServers

EthernetSwitch 1

IngestSecurity

Review WS

TapeSubsystem

Archival StorageEnclave

Fiber Channel Switches

Core Switch Backbone(w/FW + IDS + LB)

1st Floor EOP, 2nd Floor BaseFirewall + IDS

Firewall + IDS

Firewall + IDS

Firewall + IDS

EthernetSwitch

Separate Dedicated Physical Disks for Base ERA & ERA EOP

ApplicationServers

DatabaseServer

Separate Dedicated Physical Disks for Base ERA & ERA EOP

Backup/Restore Media Serversmay be separate & dedicatedbetween (Base ERA & EOP ERA)

Color = Unchanged from Base

InfrastructureAppl Server

Managem

ent LAN

M

anagement LA

N

Man

agem

ent L

AN

Firewall + IDS

EthernetSwitch 2

NetworkCompliance Mgt

Server

Ethernet Switch

ReplicaArchiveStorage

Asset MgtWorkstation

ArchiveLabel

PrintingWorkstation

Color = Changed from Base Color = New for EOP

EvolutionEvolution

An Archival Information System needs to be able to evolve in response to – Changing Information Technology

• Obsolescence

• Opportunities

– Changing business requirements

Evolution of Business Evolve

Requirements

TechnicalRequirements Solutions

Evolve Records Schedule: Current

Evolve Create Schedule Item Temporary Records Permanent Records


Openness Growth Evolution ClosureClosure

Open CloseGrow

Evolve

Open CloseGrow

Evolve ClosureClosureAn Archival Information System needs to be able to provide closure to ensure

Preservation and presentation of authentic records Comprehensive lifecycle management

of electronic records Consistency with well-established

archival science

Close ERA: a Set of Nested Systems

• Outer system lifecycle management of records of all types

• Inner Electronic Records System Ingest, preservation, disposition, and access to electronic records

• Search & Preservation Frameworks Support a variety of different approaches to different needs

• Archival “mini-systems” Specific, systematic management for each series oraggregate of electronic records

Close Document v. Record • A document is a bounded

physical representation of a body of information designedwith the capacity (and usually intent) to communicate. Adocument may manifestsymbolic, diagrammatic orsensory-representational information. ... – en.wikipedia.org/wiki/Docume

nt

• The information communicated by a document depends on itscontent and structure.

• A record is a document made or received in the course of a practical activity as aninstrument or a by-product of such activity, and set aside foraction or reference.

– http://www.interpares.org/ip2/ip2_terminology_db.cfm

• The information communicated by a record depends on itscontent, structure, and context.

Close Document

What does this document tell us?

RecordRecordClose

What does this document tell us about the U.S. Government?

Close Archival Aggregate as Directed Graph

Every record has an ‘archival bond,’ the set of relationships established by an actor between that record and other records of the actor’s activity.

Close Preservation • Documents can be preserved as individual

objects

• Records can only be preserved as ordered sets. An Archival Information System for records mustensure that

• Submission Information Packages,

• Archival Information Packages and

• Dissemination Information Packages

are managed to respect the original order of records.

Electronic RecordsClose

May be instantiated as subsets

Open

of complex ordered sets

Close ERA as a Set of Mini-Systems

A Lifecycle Management Plan for a Records Aggregate, such as a series, defines a “Mini-system;” i.e., systematic controls for that aggregate stretching from ingest to dissemination.

Open CloseGrow

Evolve SUMMARY

Openness New types of electronic records Rising and changing user expectations Creative approaches

Growth Exponential increase in volumes of data Increasing users and usage

Evolution Changing Information Technology Changing business requirements

Closure Preservation of electronic records as members of ordered sets.

Open CloseGrow

Evolve

For more information: www.archives.gov/era

Thank you.Thank you.