1

<Insert Picture Here>

Automate a Secure Historical Data Store with Oracle Total Recall

Venky Radhakrishnan Kevin JerniganDatabase Developer Senior Director Product Management

3

Oracle OpenWorld Latin America 2010

December 7–9, 2010

4

Oracle OpenWorld Beijing 2010

December 13–16, 2010

5

Oracle Products Available Online

Oracle Store

Buy Oracle license and support online today at

oracle.com/store

6

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

7

Never Lose Important Business DataTotal Recall = Total History

Why Not Track History for all Business Data

When you need to know…

How will you know?

Recreating history after the fact is expensive or impossible

When you need to know…

How will you know?

Recreating history after the fact is expensive or impossible

The ROI on “knowing who did what” can be huge

“What changes did disgruntled employee Sam make in his last week?”

The ROI on “knowing who did what” can be huge

“What changes did disgruntled employee Sam make in his last week?”

You Track History for a Few Key Fact Tables

You can answer this:

“How much of product XX was sold 3 years ago?”

Orders, Shipments

“Which employees

were on leave 3 years ago?”

“What options did product YY

have in 2001?”

“What is the average age of laptops retired

last year?”

Employees Products Assets

Can you answer these? How long would it take?

8

History is Complicated

• Keeping history is complicated• Extensive application logic is needed to track history• Must be maintained with every application change

• Viewing history is complicated• Querying and reporting history data is hard, as schemas evolve• Viewing data across multiple history tables is even harder

• The result is history is only tracked for a few key tables • Often raw fact data is tracked but context is not• e.g. Sales history is tracked, but not options, quota rules, or territories

History Kept

Orders, Shipments

Employees Products Assets

Partial or No History

9

Tracking Changes Before Total Recall

• Application logic– Problem: Direct DB updates bypass history capture

• Database triggers– Problem: Performance impact

• Log mining– Problem: Complex & error prone

10

Total Recall - Turn it on and You’re DoneTotal Recall = Total History

Transparent History Tracking Turn on archiving for the tables for which history is

not normally tracked

Total Recall automatically and transparently records

changes over time

Turn on archiving for the tables for which history is

not normally tracked

Total Recall automatically and transparently records

changes over time

Total Recall Archive

Easily Query Data as of any point in time

Not just single rows or tables, but complex joins

Easily Query Data as of any point in time

Not just single rows or tables, but complex joins

Employees Products Assets

11

Total Recall Functionality

• Automate tracking of historical database changes – Turn it on at the table level– Specify the retention period– All subsequent changes transparently archived and tamper-proof– Records older than retention period are automatically removed

• Use Flashback technologies to retrieve history– SELECT … AS OF TIMESTAMP…

SELECT … VERSIONS BETWEEN TIMESTAMP and TIMESTAMP…

• Modify schema – it keeps on working

updates history queriesBase Table

Flashback Data

Archive

Flashback Request

12

How Total Recall Works

– History captured async by background process(es) at self-tuned intervals (5 min default)

– History data stored compressed and partitioned, SecureFiles de-duplicated

– History data automatically purged per retention policy

fda1employees

Undodata

DM

LD

ML

DM

LD

ML

Originaldata in buffer

cache

Background (fda) processD

ML

DM

LD

ML

DM

L

undoundo

history

history

hist

ory

hist

ory

• Create new tablespace to hold flashback archive

• Create flashback data archive, assign to tablespace, specify retention period

• Alter base table(s) to enable archiving, assign to flashback archive

• Partitions automatically created based on time, volume

• Queries skip unrelated partitions

fda slaves (as needed). . .help

13

Total Recall Setup: 3 Easy Steps

Step 1: Create tablespace (tbs2) to hold flashback data archiveStep 2: Create flashback data archive

Step 1: Create tablespace (tbs2) to hold flashback data archiveStep 2: Create flashback data archive

fda1

CREATE FLASHBACK ARCHIVE DEFAULT fda1 TABLESPACE tbs2 RETENTION 5 YEAR;

Step 3: Enable archiving on desired tables

Step 3: Enable archiving on desired tables

employees

ALTER TABLE employeesFLASHBACK ARCHIVE fda1;

Requires new system privilege FLASHBACK ARCHIVE ADMINISTER

Requires new system privilege FLASHBACK ARCHIVE ADMINISTERGRANT FLASHBACK ARCHIVE ADMINISTER TO Bill;

Requires new object privilege FLASHBACK ARCHIVERequires new object privilege FLASHBACK ARCHIVEGRANT FLASHBACK ARCHIVE ON fda1 TO Susan;

DM

LD

ML

DM

LD

ML

historyhistory

14

Total Recall Usage: Flashback

Flashback Query (SELECT AS OF)“Retrieve the employee table as it looked 90

days ago”

Flashback Query (SELECT AS OF)“Retrieve the employee table as it looked 90

days ago”

SELECT * FROM employees AS OF (SYSDATE – 90);

Flashback Version Query (SELECT VERSIONS BETWEEN start AND end)“Retrieve Jim’s 2008 compensation history”

Flashback Version Query (SELECT VERSIONS BETWEEN start AND end)“Retrieve Jim’s 2008 compensation history”

SELECT salary, bonus FROM employee VERSIONS BETWEEN ‘2008-01-01’ and ‘2008-12-31’WHERE name = ‘Jim’;

Flashback Version Query (Pull from current and historical sources)“Get Jerry’s 401K balance since 1/1/2008”

Flashback Version Query (Pull from current and historical sources)“Get Jerry’s 401K balance since 1/1/2008”

SELECT 401_bal FROM employees VERSIONS BETWEEN ‘31-DEC-2007‘ and MAXVALUE WHERE name = ‘Jerry’;

History

employees

Base TableActive Undo

Segments

Undodata

fda1

15

TIME

What’s New in Oracle Database 11g Release 2

• Schema evolution support– Alter base table – history table automatically adjusts– Complex DDL changes (e.g. table split) accommodated

• Flashback query support across DDL changes• Performance enhancements– Parallel DML, slaves during archiving– Auto-tuning for heavy loads– Bulk archiving of small transactions

16

Transparent Schema EvolutionNew in 11g Release 2

• Schema evolution– Additional DDL support (Add Column always supported)• Drop, Rename, Modify Column• Drop, Truncate Partition• Rename, Truncate Table

– Flashback queries work across DDL changes• Output is presented accordingly

– All other DDL NOT automatically supported (see next slide)

Dro

p Co

lum

n

Add

Colu

mn

Add

Colu

mn

timetime

Flashback Version Query

17

Full Schema EvolutionNew in 11g Release 2

• Disassociate/Associate procedures– DBMS_FLASHBACK_ARCHIVE (PL/SQL package)– Disables Total Recall on specified tables, allowing more

complex DDL (upgrades, split tables, etc.)– Enforces schema integrity after association• Base table and history table schemas must be the same

– Requires FLASHBACK ARCHIVE ADMINISTER privilege

Alter schemaHistory Table

disassociate

HistoryTable

Base Table

Alter schemaassociate

18

New in Oracle Database 11.2.0.2

• Performance improvements– Faster archiving

• Support for Flashback Table

19

A Silent PartnerMinimal Overhead

Total Recall vs Triggers

6.0%8.9%

2.5%4.5%

54.0%53.2%56.9%

52.8%

0

10

20

30

40

50

60

70

100k/1k 100k/10k 1M/1k 1M/10k

# of Rows / Commit Interval

% In

crea

se in

Res

po

nse

Tim

e

Total Recall Triggers

Total Recall vs Triggers

6.0%8.9%

2.5%4.5%

54.0%53.2%56.9%

52.8%

0

10

20

30

40

50

60

70

100k/1k 100k/10k 1M/1k 1M/10k

# of Rows / Commit Interval

% In

crea

se in

Res

po

nse

Tim

e

Total Recall Triggers

5% average increase in response time with Total

Recall

( 54% average increase using Triggers )

5% average increase in response time with Total

Recall

( 54% average increase using Triggers )

20

Customer ExampleHealthways• Clinical application– OLTP app– Up to 6 concurrent batch jobs that could make more than 100K

changes per job

• Design goal: easy to maintain history/auditing system– HIPAA requirements– Fix mistakes– Debug application

• Database– 2TB, growing by 200GB/month• Growth driven by business, conversion from old system

– 5 node RAC w/ 11g Release 2, OLTP and OLAP services– Total Recall enabled on 1,026 tables– 170GB in FDA tables– Retention set to 10 years– Largest table has 26 million rows, with 42 million rows in history

21

Total Recall: A Tool for Many Uses

• ILM (Information Lifecycle Management)– Guarantee immutable history of financial data

– Automatic retention policy enforcement

• Historical reporting– Combine with DW and BI tools

– E.g Analyze product changes over time

• Error Recovery– Creates a longer flashback archive

– E.g Restore records erroneously removed or updated

22

Total Recall: A Tool for Many Uses

Combine Total Recall with Audit Vault for:

• Data forensics– E.g. Find and revert changes made by disgruntled

employee

• Employee Fraud detection– E.g Find assets that were deleted but never sold

23

Total Recall - Turn it on and You’re DoneTotal Recall = Total History

Foolproof and Secure History DataApplication TransparencyOptimal Storage FootprintEasy to Set Up Transparent History Tracking

Total Recall Archive

Employees Products Assets

24

25