ugif 12 2011-informix iwa

© 2011 IBM Corporation

Discover Informix

1

IBM Information Management

Informix Ultimate Warehouse Edition - Extreme Performance for Faster Decisions

[email protected]

Discover InformixDiscover Informix


2

The State of Data Warehouse

A Glimpse Into the Future Vendor solutions began to focus even more on the ability to isolate and prioritize workload

types including strategies for dual warehouse deployments and mixing OLTP and OLAP on the same platform.

In-memory DBMS solutions provide a technology which enables OLTP/OLAP combined solutions. Organizations should increase their emphasis on financial viability during 2011 and even into 2012 as well as aligning their analytics strategies with vendor road maps when choosing a solution.

Source: The State of Data Warehousing in 2011, 1/31/2011 by Mark Beyter, Roxane Edjlali, Donald Feinberg (ID Number: G00209643)



3

Data Warehouse Trends for the CIO, 2011-2012Data Warehouse Appliances: DW appliances are not a new concept…Most vendors have developed an

appliance offering or promote certified configurations…Although there are many reasons why organizations consider buying an appliance, the main reason is simplicity.

The Resurgence of Data Marts: Data marts can be used to optimize DW by offloading part of the workload,

returning greater performance to the warehousing environmentColumn-Store DBMSs CIOs should be aware that their current DBMS vendor may offer a column-store

solution. Don’t just buy a column-store-only DBMS because a column store was recommended by your team.

In-Memory DBMSs IMDBMS technology also introduces a higher probability that analytics and

transactional systems can share the same database.

Source: Data Warehousing Trends for the CIO, 2011-2012, 1/27/2011 by Mark Beyter, Roxane Edjlali, Donald Feinberg (ID Number: G00210272)

© 2011 IBM Corporation4

Discover Informix

IT & Business Challenges for Analytics & Data Warehouse

Costly for IT– Cost for new hardware for

processors and disks– Administering OLTP and Data

Warehouses concurrently– Expertise to tune databases

Challenges for Business– Lack of real-time operational

information– Lack of Insight from lengthy

analyses– Inability to adopt new solutions


Discover Informix

What’s New in Data Warehousing (and Analytics)?

Columnar


In-Memory Computing Technology – Defined



7

Row Oriented Data StoreEach row stored sequentially

• Optimized for record I/O • Fetch and decompress entire row, every time

• Result – • Very efficient for

transactional workloads

• Not always efficient for analytical workloads

If only few columns are required the complete row is still fetched and uncompressed



8

Columnar Data Store Data is stored sequentially by column

If attributes are not required for a specific query execution,they are skipped completely.

• Data is compressed sequentially for column:

•Aids sequential scan

•Slows random access


Discover Informix

DW Appliance, Columnar and In-Memory Databases

DW Appliance

DataAllegro (Microsoft)

Dataupia

Greenplum (EMC)

Kognito

Netezza (IBM)

Columnar DatabaseCalpont

Exasol

InfobrightParAccelSand TechnologyVertica (HP)

Sybase IQ (SAP)In-Memory OLAP Tools

QlikTech/QlikView

Applix TM-1 (IBM-Cognos)

PALO

Exalytics (Oracle)

In-Memory Data Warehouse

HANA (SAP)

ISAO-DB2 Z (IBM)

IWA (IBM)



10

Informix Warehouse Accelerator – Breakthrough Technology for Performance

1

2

34

5

6

7 1

2

34

5

6

7

Row & Columnar DatabaseRow format within IDS for transactional workloads

and columnar data access via accelerator for OLAP queries.

Extreme Compression3 to 1 compression ratio

Massive ParallelismAll cores are used for each query

Predicate evaluation on compressed data

Often scans w/o decompression during evaluation

Frequency PartitioningEnabler for the effective parallel access of

the compressed data for scanning. Horizontal and Vertical Partition

Elimination.

In Memory Database3rd generation database technology avoids I/O. Compression allows huge databases

to be completely memory resident

Multi-core and Vector Optimized Algorithms

Avoiding locking or synchronization

Comes with Smart Analytics Studio, a GUI tool, for configuring data mart and monitoring IWA



11

Informix Ultimate Warehouse Edition

What it is

*Informix Warehouse Accelerator requires a Linux Intel system as it is relies on optimizations in that environment


Discover Informix

Informix Warehouse Accelerator (Key Technologies)

Com

mon

Valu

esR

are

valu

es

Num

ber of O

ccurrenc esFrequency Partitioning

A1 D1 G1

A2 D2 G2

A4 D4 G4

A3 D3 G3

SIMD

… … … …

11111 0 1111 0

01001 0 1110 0==&

Compressed Predicate Evaluation

64-bit processor

RAM in TB

http://www.istockphoto.com/stock-photo-7893806-computer-chip.php?st=2b6ce91



13

Top 64 traded goods – 6 bit code

Rest

Prod Origin

Trade Info (volume, product, origin country)

Com

mon

Valu

esR

are

valu

es

Num

ber o f O

ccurren ces

Histogramon Origin

Histogram on Product

Origin

Prod

uct

ChinaUSA

GER,FRA,

… Rest

Table partitioned into Cells

Column Partitions

Vol

Compression: Frequency Partitioning

• Field lengths vary between cells• Higher Frequencies Shorter Codes (Approximate Huffman)

• Field lengths fixed within cells

Cell 4Cell 1

Cell 2

Cell 3

Cell 5

Cell 6



14

Data is Processed in Compressed Format

• Within a Register – Store, several columns are grouped together.

• The sum of the width of the compressed columns doesn‘t exceed a register compatible width. This utilizes the full capabilities of a 64 bit system. It doesn‘t matter how many columns are placed within the register – wide data element.

• It is beneficial to place commonly used columns within the same register – wide data element. But this requires dynamic knowledge about the executed workload (runtime statistics).

• Having multiple columns within the same register – wide data element prevents ANDing of different results.

The Register – Store is an optimization of the Column – Store approach where we try to make the best use of existing hardware. Reshuffling small data elements at runtime into a register is time consuming and can be avoided. The Register – Store also delivers good vectorization capabilities.

Predicate evaluation is done against compressed data!



15

Defining, What Data to Accelerate

• A MART is a logical collection of tables which are related to each other. For example, all tables of a single star schema would belong to the same MART.

• The administrator uses a rich client interface or SmartMart to define the tables which belong to a MART together with the information about their relationships.

• IDS creates definitions for these MARTs in the own catalog. The related data is read from the IDS tables and transferred to IWA.

• The IWA transforms the data into a highly compressed, scan optimized format which is kept locally (in memory) on the Accelerator

Define

Worker Processes

Coordinator Process

IDS + IWA

© ۲۰۱۱ IBM Corporation۱۶

Discover Informix

Informix IWA in Action At A Retail Company

Challenge Solution Result

Store Managers & Home Office Managers across thousands of stores want to analyze promotional items

Data set is ~200GB

Current database unable to provide quick enough turnaround

IWA

IWA with 24 cores single Linux Intel box

160 GB data ~ 40GB compressed RAM

< 10 secs average response with 500 users and 10x better price/performance

Able to change promotional items on a daily basis


Discover Informix

IWA in Action for Public Sector

Challenge Solution Result

Long response when police calls dispatcher

Uncoordinated data from State, County, Dept, Specialty databases

No solution offered

Informix IWA with 2 cpus & 64 GB of memory at nominal price

Seconds response time to queries

Dispatcher can provide coordinated data



18

POC with Informix Warehouse Accelerator

Data Warehouse query Performance without PerspirationAnalysis query run time reduced from 45 minutes to 4

secondsAcceleration from 60x to 1400x – average acceleration of

450xMore questions, faster answers, better business insights



19

• Microstrategy report was run, which generates

• 667 SQL statements of which 537 were Select statements• Datamart for this report has 250 Tables and 30 GB Data size• Original report on XPS and Sun Sparc M9000 took 90 mins• With IDS 11.7 on Linux Intel box, it took 40 mins• With IWA, it took 67 seconds.

POC: Datamart at a Government Agency


Discover Informix

Informix Growth Warehouse Edition

IUWE IGWE

Components Informix Ultimate Edition

Compression

IWA

ISAO Studio

Informix Growth Edition

IWA

ISAO Studio

Limits Max memory available

No core limits

Informix on 4 platforms:

AIX64, Sol64, HPUX64, Linux-ntel64

IWA on Linux-Intel 64

Informix Growth

16 cores, 16 GB Memory max

Informix on 4 platforms:

AIX64, Sol64, HPUX64, Linux-Intel64

IWA on Linux-Intel 64

48 GB Max, 16 core limit

Target > 300 GB of raw data < 300 GB of raw data

List Price $463 per PVU $150 per PVU


Discover Informix

Target Informix clients in the Ultimate Warehouse sweet spot

Informix Warehouse Editions

< 5 TB data mart

Star schema

Informix, XPS, Red Brick

MixedWorkloads

"Gemini Systems is extremely excited about the Informix Ultimate Warehouse Edition. Combining deep columnar technology with the super fast performance of in-memory databases solves many problems for both legacy and future warehouse customers. The investment preservation proposition of this offering just can't be beat. No rip-and-replace, no code rewrites, no data migrations, no tuning. Just plug-in and go for immediate business value return." - Michael "Mick" Bisignani , Senior Vice President and CTO ,Gemini Systems LLC

Informix Ultimate Warehouse Edition (IUWE) and Growth Warehouse Edition (IGWE) means higher performance and lower costs for Informix clients



22

Do you struggle with…

… performance issues on analytics and business reports ?

•Reports taking too long to run•Ad-hoc queries with unpredictable response times

… cost and flexibility for mixed workloads?•Unable to optimize on a single platform

This is an example text. Go ahead and replace it with your own text. It is meant to give you a feeling of how the designs looks including text.

… ongoing warehouse maintenance and administration?

•Constant tuning•Building/Maintaining cubes •Constant storage optimization

… leaving you at a competitive disadvantage ?

… Introducing the Informix Ultimate Warehouse Edition



23

Take “No” for an answer!

NO

Application changes

Index maintenance

Storage allocation/data

partitioning

Statistics maintenance

Cube maintenance or summary

tables

New order of Performance!

10s to 1000s of

times faster

Predictable response

times

No maintenance!!

Near zero administration!!



24

Informix Ultimate Warehouse – Performance, Simplicity, Transparency

Warehouse

Informix env Informix Warehouse Accelerator

HPUX-64, AIX-64, SOL-64, Linux-64 Linux-64, Intel

BI App

Redirect queries

Query Results

DataMart

Configure, offload data mart



25

Informix Hybrid Engine Overview

.

.

.

..



26

IWA Design Studio

DB connections

Accelerator

Workload Advisor for Mart Definition

• Takes the guesswork out of defining a data mart for IWA• Run selected queries (presumably the most time-

consuming ones) through advisor• Advisor will generate mart definition in XML format to be

loaded onto IWA• Can be fully automated



Typical Data Warehouse Architecture

All databases as marked above including OLTP, data warehouse/data mart/ODS can run on Informix



29

What Is IWA Ideally Suited For?

SALES

BRANDCATEGORY

PERIOD

PRODUCT

QUARTERMONTH

STORE

REGION

CITY

Star or snowflake schemaComplex, OLAP-style queries that typically:• Need to scan large subset of data (unlike

OLTP queries)• Involve aggregation function such as

COUNT, SUM, AVG. • Look for trends, exceptions to assist in

making actionable business decisions

SELECT PRODUCT_DEPARTMENT, REGION, SUM(REVENUE)FROM FACT_SALES F

INNER JOIN DIM_PRODUCT P ON F.FKP = P.PKINNER JOIN DIM_REGION R ON F.FKR = R.PKLEFT OUTER JOIN DIM_TIME T ON F.FKT = T.PK

WHERE T.YEAR = 2007GROUP BY PRODUCT_DEPARTMENT, REGION



T-shirt size Raw data * Main Memory Number of Intel cores (X7560)

XL >1.5 TB to 3 TB 1 TB 24-32

L >750 GB to 1.5 TB 512 20-24

M > 400 GB to 750 GB 256 GB 16-20

S > 250 GB to 400 GB 192 GB 12-16

XS ≥ 100 GB to 250 GB 96 GB 8-12XXS < 100 GB 48 GB 8

XXXS < 50 GB 24 GB 4

Sizing Guidelines

* Raw data represents only table data and excludes any indices, temp table space etc

Important ConsiderationsT-shirt sizes are a reference guideline only and are not officially available configurations.



Configuration Scenarios

Alternative 1: Install IWA on a separate Linux box

Alternative 2: Install Informix and IWA in the same symmetric multiprocessing system

Note: IWA requires Linux on Intel x64 (64-bit EM64T) Xenon

Database Server InformixWarehouse Accelerator

64 -RHEL 5,6/SUSE 1164 Solaris 10/AIX 6.1/HP-UX 11.31 -RHEL 5,6/SUSE 11

Informix Warehouse Accelerator

64-RHEL 5,6/SUSE 11

Database Server



32

The Differentiation

Deep Columnar Technology

Data is stored and accessed using columnar approach

In-Memory

Entire data set being queried is compressed and in-memory eliminating disk I/O

Run mixed workloads

OLTP transactions and OLAP queries can run against the same system

IUWE

No Maintenance

No requirements for indexes, query tuning or MOLAP cubes

ORDERS OF MAGNITUDE PERFORMANCE IMPROVEMENTS!!

1350 times

450 times

330 times

900 times The Result!!



33

Motto for UWE

“Everything should be made as simple as possible, but not simpler.”―Albert Einstein



34

Questions?contact Sandor Szabo, [email protected]

ugif 12 2011-informix iwa

Technology

column data

data warehouse appliances

ibm corporation12

ibm corporation9

ibm corporation6

ibm corporation4

ibm corporation5

state of data warehousing