1 2002: the year of archiving robert parkinson pcs limited “creating solutions through...

1

2002: The Year of Archiving

Robert Parkinson

PCS Limited“Creating Solutions Through Knowledge”

2

What We’ll Cover

Archiving benefits Definitions of terms Archiving features Archiving steps Storage and management Archiving by module What next?

3

What is Archiving?

Archiving ProgramArchiving ProgramArchive FilesArchive FilesR/3 DatabaseR/3 Database

Move data no longer required in database to archive files

Archiving Session

4

Why Archive?

Volume of data in the database increases Disk space Performance

Size of index trees, full table scans, cache hit ratios, ... Effort and resources required for database

administration Backup/restore, reorganization, create index,

statistics, ... Business and legal requirements

Auditing Record-keeping requirements Potential re-useability of data

5

Data Continues to Grow

Yearly data growth 400 GB, retention period 6 months Data not needed online after 5 years: 1,800 GB Volume without data archiving in 5 years: 1,000,000 GB Volume with data archiving in 5 years: 225,000 GB

0

250

500

750

1000

1250

1500

1750

2000

1996 1997 1998 1999 2000 2001

Not neededonline

Needed online

No backup necessary with data archiving

Backup, restore and database administrationtime grows proportional to database size withoutdata archiving

6

Archiving Requirements

Independence of database version of version and customizing of application

software of hardware (codepage / numeric format)

Auditability Revenue authorities – IRS, auditors Data security officials

Integration in applications Direct access Reporting functions

Performance and robustness

7

What We’ll Cover

Archiving benefits Definition of terms Archiving features Archiving steps Storage and management Archiving by module What next?

8

Optical Archiving

ArchiveLink can be used to archive documents

Original documents or print lists can be stored in optical archive systems and displayed, but cannot be reloaded or analyzed

Archive files which were created by R/3 data archiving can be stored, read, and reloaded using ArchiveLink

9

Database Reorganization

Should be performed after archive

Compelling reasons: Tables keep free space allocated after

archive Gives free space back to the database Improves performance Reduces database maintenance times

10

Retention vs. Residence Period

The retention period is the entire time that data spends in the database before its archived

The residence period is the minimum length of time that data must spend in the database before its eligible to be archived

11

Backup & Restore

Contains a copy of the database contents for security

Backups are usually made at regular intervals, according to a standard procedure

Reloading the backup into the database is referred to as restore

In earlier SAP releases, the archiving procedure required a database backup. SAP states this is no longer necessary in Release 3.X and 4.X

12

What We’ll Cover


13

Archiving Features

High data security during archiving Optical archive connection via ArchiveLink Data compression Online data archiving Archiving without database backup Random and sequential access to archived

data objects, reload to the database possible Automatic conversion of old archive files The ADK (Archive Developer’s Toolkit)

14

Security Provided in Archiving

Archiving is a two step process:

Create archive files

Read data from file and execute delete program

Storage of the archive files on other media

Provides data security in case of problems during the archiving process

15

Step 1 Create the Archive File

R/3 R/3 Data-Data-basebase

Archivingprogram

ArchiveArchivefilefile

16

Step 2 Execute Delete Program


Archivingprogram


Deleteprogram




Deleteprogram


Deleteprogram

17

Step 3 Store the Archive Files




Deleteprogram

•ArchiveLink•HSM-Systems

18

Optical Archive Link

ArchiveLink The archive files can be stored on

tertiary storage media Files can be stored manually or

automatically

Hierarchical storage management systems (HSM systems)

Archive files can also be passed to an HSM system

HSM systems can use optical and optomagnetic storage media

The HSM system manages the archive files automatically

ArchiveLink

R/3 System

Archive system

19

Compression

Data is automatically compressed (Up to a factor of 5 or greater)

Online data archiving

Data archiving may run in parallel to dialog and batch processing

Data archiving without database backup

Data archiving is fully recoverable

20

Access to Archived Data

There are two ways to access archived data:

Sequential access to a dataset (reporting) Random access to individual data objects

Archive files can be

Read Analyzed (for reporting purposes) Reloaded (Not Recommended – only right after

archiving!) Actions depend on archiving object

Access for reporting and reloading requires that the file is in the file system

Access to individual objects depends on an index

21

Conversion of Old Archive Files

Necessary conversions are performed automatically

Temporary conversion during data access

In rare cases, conversion programs are provided

22

The ADK

Database

R/3 System

LDB

ABAP/4 Program ArchivingClass

Codepage, Number Format, Structure Conversion Direct Access, Compression, Batch-Handling, . . . .

ADK:

Archive File

manually HSM

Archive System with Tertiary Storage Media

ArchiveLink

23

What is an Archiving Object?

R/3 SystemR/3 System

DatabaseDatabase

ArchiveArchive

Key to archiving concept Description of database objects Collection of programs

24

Customization During Data Archiving

Creating an archiving object Define archive object using transaction AOBJ Create programs for archiving

Archive program Delete program Reload program Index program

Create programs for accessing archived data Reports Direct access via online transaction

25

What We’ll Cover


26

Archiving Personnel

Archiving can be performed by the Basis Administrator by the application department

Motivation (and point of entry) for both are different

Basis Administrator The Basis Administrator checks the database/table size Archiving object relationship has to be determined

Application Dept. Application Dept. performs archiving for various reasons

(periodically – month end, quarter end, year end) Application Dept. uses direct point of entry

27

Archiving Flowchart

C o o rd in a te a c tiv it ies

C h eck de p en d e nc ies

B e fore 1 s t u se o f a rch iv ing


C h eck de p en d e nc ies

C h e ck a p p lica tio n cu s to m iz ing

P la tfo rm in de p en d en t f ile na m e

A rch ive f ile s ize

A rch ive L ink

D e le te p ro g ram

C h e ck A D K cu s tom iz ing

B e fore 1s t u se o f p a rticu la r a rch iv ing ob je ct


C h e ck sp ace on h ard d isk

S ch ed u le a rch iv ing sess ion

C h e ck a rch iv in g re su lts

S a ve a rch ive f iles

F o r e ach se ss ion

A rch iv in g p ro ced u res

28

Identifying Critical Tables

Database Performance Screen (DB02) Overall database growth analysis Determine size of tablespaces and tables Free storage space – allocated vs used space

Table Call Statistics Screen (ST10) Locate the most active tables Which tables are growing Which tables have slow response

29

Identifying Critical Archiving Objects

Use archive administration (SARA) Browse through the archiving objects Determine which tables are in the archiving

objects Schedule all required archive programs

Use CCMS database administration (DB15) Determine which archiving objects use a table Check online space information for all tables of

an archiving object

30

Business Requirements for Archiving Objects (SD_VBAK)

Document has status "Complete" All documents in document flow

have status "Complete" Financial document is balanced Additional customized checks

possible

31

What We’ll Cover


32

Storage & Management

Automated storage Low management effort

Manual storage Data medium maintenance Data medium management Archive production

33

Storage of Archive Files

Choice of medium is critical for: Utilization of the functionality offered by SAP Effort required for maintenance Effort required to access data Durability of archive

Protection of investment by means of combined methodology Data archive files Workflow for incoming and outgoing records DMS (Document Mgmt. System) technical documents Printed reports Automation of database backup and restore

34

Archive Files on Hard Disk

Advantages Single document access < 5 seconds Quick access for reporting

Disadvantages File system limitations Considerable manual management effort

required Replicate the archive files Storage of archive files Data security

35

Archive Files on HSM System

Advantages Single document access < 30 seconds (with

block access) Quick access for reporting (with block

access) Scalable file system Long data medium retention period Data medium management support

Disadvantages Can be a costly solution

36

HSM System Requirements

"Write through" should be possible Increased data security, as the archive files

migrate before the delete program runs Data retained in "magnetic cache" as copy

"Random access" to files must be possible "Hardware sharing" is useful

Use of a jukebox via both HSM and ArchiveLink "Media management" should be supported

Replicate archive files Store archive files

37

Archive Files on Jukebox via ArchiveLink

Advantages Scalable system Long data medium retention period Data medium management support Jukebox can also be used for other R/3 tasks

Disadvantages Can be a costly solution Single document access only feasible for

small archive files Access time heavily depends on archive file

size

38

ArchiveLink Capabilities

ArchiveLink

R/3 System

Archive system

With ArchiveLink you can: Archive incoming R/3 documents

Archive information created in R/3

Directly access the optically archived documents from the R/3 applications

Integrate an optical archive system with R/3

Replace your existing paper archive with electronic archive systems

39

Some Considerations for Media

Jukebox via ArchiveLink Archived data seldom accessed individually Access time is not critical, random access slow in Rel.

< 3.1G HSM system with block access

Archived data are often accessed individually Access time can be on the order of 30 seconds

Hard disk Archived data are regularly accessed individually Access time must be on the order of 5 seconds

Magnetic tape Archived data are only accessed in exceptional cases

40

Sample Access Times for Archived Data

25 MB 50 MB 100 MB0

30

60

90

120

150

180

210

240

270

300

25 MB 50 MB 100 MB

Hard Disk

HSM

ArchiveLinkrandomaccess

ArchiveLinkreporting

Sec.

41

What We’ll Cover


42

FI Documents

600,000 FI documents per month Retention period up to 18 months 120 GB used space in DB for FI data 100,000 documents archived in 3

hours, 80 MB archive file Reduce retention period to 3 months Data archiving by IT organization

43

MM Documents

450,000 MM documents and 2,500,000 line items per month

Line-item table bigger than 10 GB 100,000 line items being archived in 3

hours Direct access implemented by customer Reduce retention period Random access and reporting on data

archive files shows good performance

44

Access Times … Example

Optimal situation, single job, elapsed timeArchiving Object create 1000 archive 1000

FI_DOCUMNT 500s 60s

SD_VBAK 1000s 160s

SD_VBRK 1000s 140s + 60s

RV_LIKP 1000s 340s

MM_MATBEL 1000s 60s + 60s

45

What We’ll Cover

Archiving benefits Definition of terms Archiving features Archiving steps Storage and management Archiving by module What Next?

46

What next?

Increase in third party solutions Better reporting for archiving

objects More robust document imaging

solutions Enhanced performance of ADK Further integration of archiving and

database administration

47

Sources of Information

Check OSS for your platform SAPNET good source of information SAPLabs, SAP Search, mySAP.com SAP’s Early Watch for data

archiving https://www014.sap-ag.de/dma

[email protected]

48

Your Turn! (to launch Q&A)

PCS Limited, A Professional PCS Limited, A Professional IT Consulting Services IT Consulting Services

Company Company ““Creating Solutions Through Knowledge” Creating Solutions Through Knowledge”

PCS LimitedPCS Limited

5425 Parkford Circle5425 Parkford CircleGranite Bay, CA 95746 USA Granite Bay, CA 95746 USA

Tel 916.773.9991Tel 916.773.9991Fax 916.773.9998 Fax 916.773.9998

www.pcs-limited.net www.pcs-limited.net [email protected]@pcs-limited.net