1. automate a secure historical data store with oracle total recall venky radhakrishnankevin...
Post on 28-Dec-2015
221 Views
Preview:
TRANSCRIPT
<Insert Picture Here>
Automate a Secure Historical Data Store with Oracle Total Recall
Venky Radhakrishnan Kevin JerniganDatabase Developer Senior Director Product Management
5
Oracle Products Available Online
Oracle Store
Buy Oracle license and support online today at
oracle.com/store
6
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
7
Never Lose Important Business DataTotal Recall = Total History
Why Not Track History for all Business Data
When you need to know…
How will you know?
Recreating history after the fact is expensive or impossible
When you need to know…
How will you know?
Recreating history after the fact is expensive or impossible
The ROI on “knowing who did what” can be huge
“What changes did disgruntled employee Sam make in his last week?”
The ROI on “knowing who did what” can be huge
“What changes did disgruntled employee Sam make in his last week?”
You Track History for a Few Key Fact Tables
You can answer this:
“How much of product XX was sold 3 years ago?”
Orders, Shipments
“Which employees
were on leave 3 years ago?”
“What options did product YY
have in 2001?”
“What is the average age of laptops retired
last year?”
Employees Products Assets
Can you answer these? How long would it take?
8
History is Complicated
• Keeping history is complicated• Extensive application logic is needed to track history• Must be maintained with every application change
• Viewing history is complicated• Querying and reporting history data is hard, as schemas evolve• Viewing data across multiple history tables is even harder
• The result is history is only tracked for a few key tables • Often raw fact data is tracked but context is not• e.g. Sales history is tracked, but not options, quota rules, or territories
History Kept
Orders, Shipments
Employees Products Assets
Partial or No History
9
Tracking Changes Before Total Recall
• Application logic– Problem: Direct DB updates bypass history capture
• Database triggers– Problem: Performance impact
• Log mining– Problem: Complex & error prone
10
Total Recall - Turn it on and You’re DoneTotal Recall = Total History
Transparent History Tracking Turn on archiving for the tables for which history is
not normally tracked
Total Recall automatically and transparently records
changes over time
Turn on archiving for the tables for which history is
not normally tracked
Total Recall automatically and transparently records
changes over time
Total Recall Archive
Easily Query Data as of any point in time
Not just single rows or tables, but complex joins
Easily Query Data as of any point in time
Not just single rows or tables, but complex joins
Employees Products Assets
11
Total Recall Functionality
• Automate tracking of historical database changes – Turn it on at the table level– Specify the retention period– All subsequent changes transparently archived and tamper-proof– Records older than retention period are automatically removed
• Use Flashback technologies to retrieve history– SELECT … AS OF TIMESTAMP…
SELECT … VERSIONS BETWEEN TIMESTAMP and TIMESTAMP…
• Modify schema – it keeps on working
updates history queriesBase Table
Flashback Data
Archive
Flashback Request
12
How Total Recall Works
– History captured async by background process(es) at self-tuned intervals (5 min default)
– History data stored compressed and partitioned, SecureFiles de-duplicated
– History data automatically purged per retention policy
fda1employees
Undodata
DM
LD
ML
DM
LD
ML
Originaldata in buffer
cache
Background (fda) processD
ML
DM
LD
ML
DM
L
undoundo
history
history
hist
ory
hist
ory
• Create new tablespace to hold flashback archive
• Create flashback data archive, assign to tablespace, specify retention period
• Alter base table(s) to enable archiving, assign to flashback archive
• Partitions automatically created based on time, volume
• Queries skip unrelated partitions
fda slaves (as needed). . .help
13
Total Recall Setup: 3 Easy Steps
Step 1: Create tablespace (tbs2) to hold flashback data archiveStep 2: Create flashback data archive
Step 1: Create tablespace (tbs2) to hold flashback data archiveStep 2: Create flashback data archive
fda1
CREATE FLASHBACK ARCHIVE DEFAULT fda1 TABLESPACE tbs2 RETENTION 5 YEAR;
Step 3: Enable archiving on desired tables
Step 3: Enable archiving on desired tables
employees
ALTER TABLE employeesFLASHBACK ARCHIVE fda1;
Requires new system privilege FLASHBACK ARCHIVE ADMINISTER
Requires new system privilege FLASHBACK ARCHIVE ADMINISTERGRANT FLASHBACK ARCHIVE ADMINISTER TO Bill;
Requires new object privilege FLASHBACK ARCHIVERequires new object privilege FLASHBACK ARCHIVEGRANT FLASHBACK ARCHIVE ON fda1 TO Susan;
DM
LD
ML
DM
LD
ML
historyhistory
14
Total Recall Usage: Flashback
Flashback Query (SELECT AS OF)“Retrieve the employee table as it looked 90
days ago”
Flashback Query (SELECT AS OF)“Retrieve the employee table as it looked 90
days ago”
SELECT * FROM employees AS OF (SYSDATE – 90);
Flashback Version Query (SELECT VERSIONS BETWEEN start AND end)“Retrieve Jim’s 2008 compensation history”
Flashback Version Query (SELECT VERSIONS BETWEEN start AND end)“Retrieve Jim’s 2008 compensation history”
SELECT salary, bonus FROM employee VERSIONS BETWEEN ‘2008-01-01’ and ‘2008-12-31’WHERE name = ‘Jim’;
Flashback Version Query (Pull from current and historical sources)“Get Jerry’s 401K balance since 1/1/2008”
Flashback Version Query (Pull from current and historical sources)“Get Jerry’s 401K balance since 1/1/2008”
SELECT 401_bal FROM employees VERSIONS BETWEEN ‘31-DEC-2007‘ and MAXVALUE WHERE name = ‘Jerry’;
History
employees
Base TableActive Undo
Segments
Undodata
fda1
15
TIME
What’s New in Oracle Database 11g Release 2
• Schema evolution support– Alter base table – history table automatically adjusts– Complex DDL changes (e.g. table split) accommodated
• Flashback query support across DDL changes• Performance enhancements– Parallel DML, slaves during archiving– Auto-tuning for heavy loads– Bulk archiving of small transactions
16
Transparent Schema EvolutionNew in 11g Release 2
• Schema evolution– Additional DDL support (Add Column always supported)• Drop, Rename, Modify Column• Drop, Truncate Partition• Rename, Truncate Table
– Flashback queries work across DDL changes• Output is presented accordingly
– All other DDL NOT automatically supported (see next slide)
Dro
p Co
lum
n
Add
Colu
mn
Add
Colu
mn
timetime
Flashback Version Query
17
Full Schema EvolutionNew in 11g Release 2
• Disassociate/Associate procedures– DBMS_FLASHBACK_ARCHIVE (PL/SQL package)– Disables Total Recall on specified tables, allowing more
complex DDL (upgrades, split tables, etc.)– Enforces schema integrity after association• Base table and history table schemas must be the same
– Requires FLASHBACK ARCHIVE ADMINISTER privilege
Alter schemaHistory Table
disassociate
HistoryTable
Base Table
Alter schemaassociate
18
New in Oracle Database 11.2.0.2
• Performance improvements– Faster archiving
• Support for Flashback Table
19
A Silent PartnerMinimal Overhead
Total Recall vs Triggers
6.0%8.9%
2.5%4.5%
54.0%53.2%56.9%
52.8%
0
10
20
30
40
50
60
70
100k/1k 100k/10k 1M/1k 1M/10k
# of Rows / Commit Interval
% In
crea
se in
Res
po
nse
Tim
e
Total Recall Triggers
Total Recall vs Triggers
6.0%8.9%
2.5%4.5%
54.0%53.2%56.9%
52.8%
0
10
20
30
40
50
60
70
100k/1k 100k/10k 1M/1k 1M/10k
# of Rows / Commit Interval
% In
crea
se in
Res
po
nse
Tim
e
Total Recall Triggers
5% average increase in response time with Total
Recall
( 54% average increase using Triggers )
5% average increase in response time with Total
Recall
( 54% average increase using Triggers )
20
Customer ExampleHealthways• Clinical application– OLTP app– Up to 6 concurrent batch jobs that could make more than 100K
changes per job
• Design goal: easy to maintain history/auditing system– HIPAA requirements– Fix mistakes– Debug application
• Database– 2TB, growing by 200GB/month• Growth driven by business, conversion from old system
– 5 node RAC w/ 11g Release 2, OLTP and OLAP services– Total Recall enabled on 1,026 tables– 170GB in FDA tables– Retention set to 10 years– Largest table has 26 million rows, with 42 million rows in history
21
Total Recall: A Tool for Many Uses
• ILM (Information Lifecycle Management)– Guarantee immutable history of financial data
– Automatic retention policy enforcement
• Historical reporting– Combine with DW and BI tools
– E.g Analyze product changes over time
• Error Recovery– Creates a longer flashback archive
– E.g Restore records erroneously removed or updated
22
Total Recall: A Tool for Many Uses
Combine Total Recall with Audit Vault for:
• Data forensics– E.g. Find and revert changes made by disgruntled
employee
• Employee Fraud detection– E.g Find assets that were deleted but never sold
23
Total Recall - Turn it on and You’re DoneTotal Recall = Total History
Foolproof and Secure History DataApplication TransparencyOptimal Storage FootprintEasy to Set Up Transparent History Tracking
Total Recall Archive
Employees Products Assets
top related