sap sybase iq15 vldb option technical overview courtney claussen analytics product management team...
TRANSCRIPT
![Page 1: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/1.jpg)
SAP SYBASE IQ15 VLDB OPTION
TECHNICAL OVERVIEW
COURTNEY CLAUSSENANALYTICS PRODUCT MANAGEMENT [email protected], 2012
![Page 2: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/2.jpg)
• Product Success• What is Information Lifecycle Management?• SAP SYBASE IQVLDB Option• VLDB in Use in a Large Bank• PowerDesigner ILM Model for SAP SYBASE IQ• Summary
AGENDA
![Page 3: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/3.jpg)
PRODUCT SUCCESS
![Page 4: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/4.jpg)
Mature, industrial strength analytic DBMS
SAP SYBASE IQ
LEADERSHIP
• Industry leading performance & scale benchmarks
• Recognized EDW market leader by Gartner, Forrester
• Pioneering technology with 10+ patents
ADOPTION
• 4500+ installations in 2150+ accounts
• ~200 new customer wins per year (last 4 years)
• Consistently 96%+ customer satisfaction rates
MOMENTUM
• 2 x DW market growth rate (last 4 years)
• Fast paced product releases
• v15, v15.1 (2009), v15.2 (2010), v15.3, v15.4 (2011)
Ericsson • Sungard • Nielsen • BNP Paribas • Telefonica • hmv.com • comScore • Agricultural Bank of China
![Page 5: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/5.jpg)
Stores and analyzes large amounts of data
SAP SYBASE IQ
Stands out as the leading enterprise data warehouse amongst the largest banks, insurance agencies, and telecom operators worldwide
Manage and analyze statistical measures for the entire nation of Canada
Analyze complex models in more than 200 financial institutions worldwide
Analyze ALL Federal tax returns in the US
Store and Analyze massive amounts of industry segment data in 30 of the largest information providers in the world, including Transunion, Nielsen and Axiom
![Page 6: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/6.jpg)
WHAT IS INFORMATION LIFECYCLE MANAGEMENT?
![Page 7: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/7.jpg)
“ ILM is a management approach aimed at tackling the storage ‘information overload' problem which has so far failed to live up to its potential. The key to its success is being able to automate identification of the most valuable information contained in company data at any given time so that relatively unimportant data can be automatically demoted to lower-cost, less accessible storage media and ultimately discarded.”
BLOOR RESEARCH
![Page 8: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/8.jpg)
NOAA: National Oceanic and Atmospheric Administration
• A global network of sensors provide a steady stream of data on the Earth’s oceans and weather• With streams and a vast archive of historical data, NOAA
manages some of the largest databases in federal government• The Princeton, NJ data center alone stores more than 20
petabytes of data• NOAA CIO: Joe Klimavicz:– “I focus much of my time on DATA LIFECYCLE MANAGEMENT– “The keys to ensuring that data is useable and easy to find
include using accurate metadata, publishing data in standard formats, and having a WELL-CONCEIVED DATA STORAGE STRATEGY”
ILM IN THE REAL WORLD
![Page 9: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/9.jpg)
Data lifecycle
DATA DECREASES IN VALUE OVER TIME
Months
Business E
vent
Operational Transaction
Data Transform
and Load into DW
Data is Querie
d, Analysed and Reported
Data is Archived
Data is Purged
TimeHour/s Day/sMinute/s Year/s Decade/sT=0
![Page 10: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/10.jpg)
4. Drop partition
JanFebMarAprMayJun1. Roll-on:
Load monthlytable partition
2. Mark partition read-only
Dec
SepAug
JulJun
5. Drop backup files
Data Partitions
Data partitioning and placement according to data value
INFORMATION LIFECYCLE MANAGEMENT
3. Back-up the partition
![Page 11: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/11.jpg)
SAP SYBASE IQVLDB OPTION
![Page 12: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/12.jpg)
SAP SAP SYBASE IQ15 Engine
Multiplex G
rid Architecture
Admin &
Monitoring Fram
ework
Storage Area Network
Communications &Security
Column IndexingSub-system
LoadingEngine
Column StorageProcessor
QueryEngine
In-Database Analytics
Text Search
Web Enabled Analytics
Information Lifecycle M
anagement
Manage data through its existence in the DW
Information lifecycle management
SAP SYBASE IQ
![Page 13: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/13.jpg)
• Data partitioning• Multiple user DBSpaces• Separate unstructured data from transactional data• Place frequently accessed data on fast storage• Granular database administration with read-only, read-write, on-line
and off-line DBSpaces
SAP SYBASE IQVLDB OPTION
Catalog StoreIQ Main Store for
User Data Temp Store
Table
DBSpace
DBFile
Table Partition
DBSpace
DBFile
Table Column
DBSpace
DBFile
Index
DBSpace
DBFile
![Page 14: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/14.jpg)
OptionPartitioned Tables
Number of User DBSpaces
Database Object Placement
DBSpace Attributes
DBSpace Management
VLDB Option
• Partition by range; single column partition key
•Multiple DBSpaces, each with multiple DBFiles• Unlimited data
volume
• Place database objects (tables, table partitions, columns, indexes) in specific DBSpaces
• DBSpaces can be marked read-only, read-write, on-line or off-line
• Validate read-write portions of database separately from read-only• Backup read-
write DBSpaces separately from read-only
SAP SAP SYBASE IQBase Product
• Single table partition
• Single user DBSpace with multiple DBFiles• Unlimited data
volume
• All database objects are placed in one user DBSpace
• Single user DBSpace is read-write and on-line
• Validate and backup single user DBSpace as a unit
Benefits
VLDB OPTION
![Page 15: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/15.jpg)
Partitioning and placement
• IQ provides partitioning and placement features to manage the storage and movement of data:– Partitioning divides data into non-overlapping subsets across
a dimension, such as “date”. For example, you may partition customer order data by date– Placement maps a data partition to a particular area of
storage: the partition “June Customer Orders 2009” resides in file “/opt/data/orders/june2009.dat”
• Separate big, unstructured data from transactional data:– Different levels of protection– Different administration needs– Use of tiered storage to control cost
ILM IN SAP SYBASE IQ
![Page 16: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/16.jpg)
• Database administrative operations can be performed with finer control– The database can be divided into read-only and read-write
sections that are managed differently Backup and restore time can be reduced by backing up read-only
data once Data validation can be invoked on just the read-write portions of the
database
– Frequently accessed data can be assigned to faster data storage, and less frequently accessed data can be segregated to cheaper, slower storage
CONTROLS FOR DATABASE ADMINISTRATION
![Page 17: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/17.jpg)
Partition by range: single column partition key
PARTITION AND POSITION A TABLE IN IQ
CREATE TABLE Orders (OrderID INT,OrderDate DATE,Description CHAR(10) ,PARTITION BY RANGE (OrderDate( p2010 VALUES < ='2010-12-31‘ IN FIBER,p2011 VALUES <= '2011-12-31‘ IN FIBER, pNextYear VALUES <= (MAX) IN FIBER);
1) Partition table Orders
Over time, as data is being loaded, start migratingolder data to slower, cheaper storage
ALTER TABLE Orders DROP PARTITION p2010;
2) Move p2010 to SATA storage
ALTER TABLE Orders MOVE PARTITION p2010 to SATA;
3) Later, drop very old partitions
![Page 18: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/18.jpg)
Virtual data mart of servers and partitioned storage • Workload management• Privacy through isolation of resources• Separate big unstructured data from transactional data• Back up and restore independently
Unique, user community focused platform for big data analytics
VIRTUAL DATA MARTS
Data Scientists Business Analysts Operations End Users
Full Mesh High Speed InterconnectFull Mesh High Speed Interconnect
SAN Fabric
![Page 19: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/19.jpg)
VLDB IN USE AT A LARGE BANK
![Page 20: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/20.jpg)
SHORTEN DATA BACKUP TIMES
![Page 21: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/21.jpg)
RE-CLAIM VALUABLE STORAGE SPACE
![Page 22: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/22.jpg)
POWERDESIGNER ILM MODEL FOR SAP SYBASE IQ
![Page 23: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/23.jpg)
• Model the database• Create DBSpaces– Assign cost
• Create a new lifecycle– Assign start date and phase retention periods
• Associate tables with lifecycle– Select date column partition key
• Estimate cost savings• Generate scripts to move partitions through DBSpaces as they
age
ILM IN POWERDESIGNER
![Page 24: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/24.jpg)
CREATE LIFECYCLE
![Page 25: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/25.jpg)
LIFECYCLE PROPERTIES
Assign a cost to the storage:
Indicate which tables are part of the lifecycle:
![Page 26: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/26.jpg)
GENERATE DATA MOVEMENT SCRIPTS
![Page 27: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/27.jpg)
GENERATE COST SAVINGS REPORT
Generate cost savings information
Report:
![Page 28: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/28.jpg)
SUMMARY
![Page 29: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/29.jpg)
SUMMARY
• Storage strategies for managing big data — to service data requests responsively, while controlling costs• Learn more– Visit: http://www.sybase.com/sybaseiq-vldb – Call: 1.800.792.2735
SAP SYBASE IQVLDB OPTION
![Page 30: SAP SYBASE IQ15 VLDB OPTION TECHNICAL OVERVIEW COURTNEY CLAUSSEN ANALYTICS PRODUCT MANAGEMENT TEAM COURTNEY.CLAUSSEN@SAP.COM FEBRUARY, 2012](https://reader030.vdocuments.mx/reader030/viewer/2022020115/5519b4a65503465b578b473a/html5/thumbnails/30.jpg)