Next Generation BI and Data Warehouse Solutions

Ross LoForteSQL Technology ArchitectMicrosoft Technology Centers

Data Quality

Real-Time DW and Streaming Data

Advanced Analytics

MPP

MDM

Secure and Robust

Key TrendsMPP

(Parallel Data

Warehouse)

Master Data

Services

Database Security

StreamInsight

(Streaming Data)

Data Quality (Zoomix)

Data Warehouse Industry TrendsMicrosoft has steadily invested in the most important data warehouse

technologies

Column Store

Column Store

(Project Apollo)

Microsoft’s on-going investments in Data Warehousing

Heterogeneous Connectivity & Workloads

Data Integrity & Quality

Compliance & Security

Data Warehouse Scale

Data Warehouse Management

2005 2008 Futures

PB Warehouses>64 Core ProcessingScale out through MPP

Perf. Management ToolsBI Resource GovernanceImproved Predictability

Mixed workload supportContinuous Loading

Master Data Management(Stratature Integration)Integrated DQ Services (Zoomix)

Rights Management

10s of TB WarehousesParallel partitioningData compressionNew Reference

Architectures

Policy Based Admin.DB Resource

Governance

High Perf. Connectors(Oracle, Teradata, SAP BW)

Data Profiling

Policy based auditing

Multi TB WarehousesEnterprise scalabilityDW Reference

Architectures

Unified manageability

Enterprise class ETL tool

Data Cleansing(Fuzzy lookup/matching)

Data Protection & Tracing

SQL Server Top Achievements

Category MetricLargest single database 70 TBLargest table 20 TB

Biggest total data 1 application

88 PB

Highest database transactions per second 1 db (from Perfmon)

130,000

Fastest I/O subsystem in production (SQLIO 64k buffer)

18 GB/sec

Fastest “real time” cube 5 sec latency

Data load for 1TB 20 minutesLargest single cube 12 TB

Microsoft Data Warehousing solutions

Tier 1 offerings

Tier 1 Services and Support

Scalable and reliable platform for Data

Warehousing on any hardware

Reference Architectures offering best price

performance for data warehousing

Scalable and reliable platform for Data

Warehousing on any hardware

Appliance for high end Data Warehousing requiring highest

scalability, performance or complexity

Ideal for data marts or small to mid-sized EDWs

Ideal for data marts or small to mid-sized DWs

with scan centric workloads

Ideal for large data marts or mid-sized EDWs

Offers flexibility in hardware and architecture

Software only Reference Architectures (Software and Hardware)

Software onlyDW Appliance

(Fully integrated Software and Hardware)

Scale-Up DW Scale-Up DW Scale-Up DW Scale-Out DW with MPP

10s of TB 4 – 80 TB 10s of TB 10s - 100s of TB

$28.8K/Proc$9.9K/Svr + $162/CAL

$107K - $683K (2 – 8 Procs; includes

Hardware)$57.5K/Proc only $38.3K/Proc

Microsoft Data Warehousing solutions

Integrated ETL and Reporting toolsSimplified managementPredictable responseLower storage costsIntegrated Master Data Management tool

Tier 1 offerings

Scalable and reliable platform for Data

Warehousing on any hardware

Ideal for data marts or small to mid-sized EDWs

Software only

Scale-Up DW

10s of TB

$28.8K/Proc$9.9K/Svr + $162/CAL

Microsoft Data Warehousing solutions

All features and benefits of SQL Server 2008 R2 Enterprise Ability to scale up to 256 logical processorsAbility to scale memory beyond 2TBContinuous loading using StreamInsight

Tier 1 offerings

Scalable and reliable platform for Data

Warehousing on any hardware

Ideal for large data marts or mid-sized EDWs

Software only

Scale-Up DW

10s of TB

$57.5K/Proc only

Microsoft Data Warehousing solutions

Balanced solution for scan-centric workloadsBest price-to-performance ratioFeatures 12 reference architectures validated by MicrosoftAbility to scale up to 80 terabytes

Tier 1 offerings

Reference Architectures offering best price

performance for data warehousing

Ideal for data marts or small to mid-sized DWs

with scan centric workloads

Reference Architectures (Software and Hardware)

Scale-Up DW

4 – 80 TB

$107K - $683K (2 – 8 Procs; includes

Hardware)

Some Data Warehouses Today

Big SANBig SMP ServerConnected together

• Server can consume 32 GB/Sec of IO, but SAN can only deliver 12 GB/Sec

• Queries are slow− Despite significant investment in both Server and Storage

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

Challenges of traditional Data Warehouse

CPU

IO Channe

l

CPU Constraint

Sequential IO

capacity of

storage System

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

Sequential IO

capacity of

storage System

CPU

Storage System Constraint

IO Channe

l

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101

IO Chann

el

Sequential IO

capacity of storage

System

IO Channel constraint

CPU

What is Fast Track Data Warehouse?• A method for designing a cost-effective, balanced

system for Data Warehouse workloads • Reference hardware configurations developed in

conjunction with hardware partners using this method

• Best practices for data layout, loading and management

Relational Database Only – Not SSAS, IS, RS

Fast Track SQL DW Architecture vs. Traditional DW

SQL 2008 Data Warehouse4 Processor 16 Core Server

Shared Network Bandwidth

Enterprise Shared SAN Storage

Dedicated Network Bandwidth

Traditional SQL DWArchitectureShared Infrastructure

Fast Track SQL DW ArchitectureDedicated DW InfrastructureArchitecture modeled after DW Appliances 1TB – 80TB Pre-Tested

Dedicated Low Cost SAN Arrays 1 for every 4 CPU Cores HP MSA2312

OLTP Applications

Benefits:-More System Predictability Thus User Experience-Pretested Configurations Lowers TCO-Balanced CPU to I/O Channel Optimized for DW-Modular Building Block Approach-Scale Out or Up within limits of Server and SAN

Reference architectures boost performance and reduce risk

HP Fast Track data warehouse configurations scale from SMB to Enterprise

• Prescriptive guidance and optimized methodology for data warehouse query workloads with large sequential data reads

• Balanced hardware approach ideal for data marts or small to mid-sized DW with scan-centric workloads

• Supports 1 to 80TB Data Warehouse at leading price/performance

• Configurations, tested performance guidance and best practices for deploying/operating/managing

• Packaged and custom support

Basic6 – 12TBDL38x w/

MSA P2000

Mainstream12 – 24TBDL585 w/

MSA P2000

Mainstream16 – 32 TB DL580 w/

MSA P2000

Premium24 – 80 TBDL980 w/

MSA P2000

Entry1-5TBDL370

w/D2700 DAS

DemoFast Track Data Warehouse

Microsoft Data Warehousing solutions

Enterprise Data Warehouse Appliance offeringHigh Scalability and performanceFlexibility and choiceIntegrated with Microsoft BI

Tier 1 offerings

Appliance for high end Data Warehousing requiring highest

scalability, performance or complexity

Offers flexibility in hardware and architecture

DW Appliance(Fully integrated

Software and Hardware)

Scale-Out DW with MPP

10s - 100s of TB

$38.3K/Proc

• All hardware from a single vendor• Orderable at the rack level• Vendor will:

− Assemble appliances− Image appliances with OS, SQL

Server, and PDW software• Appliance installed in 1 – 2 days• Support:

− Microsoft provides first call support− Hardware partner provides onsite

break/fix support

Parallel Data WarehouseAn appliance experience

Control Rack Data Rack

Compute Nodes Storage Nodes

Spare Compute

Node

Du

al

Fib

er

Ch

an

nel

SQL

SQL

SQL

SQL

SQL

SQL

SQL

SQLDu

al

Infi

nib

an

d

Control Nodes

Active /

Passive

Landing Zone

Backup node

SQL

Management Node

SQL

SQL

DemoParallel Data Warehouse

Admin Console – Home Page

• Menu options listed left to right by PDW activity and status.

Admin Console – Appliance State

• Appliance State tab lists the state of all active nodes within the appliance.

Admin Console – Dashboard Customizations• Can optionally include up to 38 available

performance counters.

…

Admin Console – Dashboard

• The Dashboard tab provides near real-time performance counters.

23

Distributed Data Warehouse Architecture

• Each business unit has own Data Marts− More responsive to business needs− Fits budget realities

• Hub provides centralized data governance etc.• Node-to-node data movement

− Parallel over Infiniband− >500GB per min− Parallel Database Export (PDE)

Delivered through a Familiar Interface• Self-Service access

& insight• Data exploration

& analysis• Predictive analysis• Data visualization• Contextual

visualization

The Microsoft BI Solution Stack

BUSINESS COLLABORATION PLATFORM

DATA INFRASTRUCTURE & BUSINESS INTELLIGENCE PLATFORM

BUSINESS USER EXPERIENCE

Business Productivity Infrastructure• Dashboards &

Scorecards• Excel Services• Web based forms

& workflow• Collaboration• Search• Content

Management• LOB data integration• PowerPivot for

SharePoint

The Microsoft BI Solution Stack

BUSINESS COLLABORATION PLATFORM

DATA INFRASTRUCTURE & BUSINESS INTELLIGENCE PLATFORM

BUSINESS USER EXPERIENCE

Data Infrastructure & BI Platform• Analysis Services• Reporting Services• Master Data Services• Integration Services• Data Mining• Data Warehousing

BUSINESS COLLABORATION PLATFORM

DATA INFRASTRUCTURE & BUSINESS INTELLIGENCE PLATFORM

BUSINESS USER EXPERIENCE

The Microsoft BI Solution Stack

Use Reports to Drive Decisions

• Create and share reports• Maintain a single version of truth with your Excel

Workbooks• Drive decision based on facts

Use Dashboards to Drive Decisions

• Visual displays of information needed to achieve one or more objectives

• Single-Screen display of information

• Answer fundamental questions

• Alerts the user to issues or problems

• Span Operational, Performance, Personal

• Align strategies and organizational goals

• Measure and manage Key Performance Indicators (KPI)

• Modeled after the business, not the data

PowerPivot for Excel PowerPivot for SharePoint

Use PowerPivot to Drive Self-Services

29

Microsoft Business Decision Appliance

• Rich insight: Empower users to easily create PowerPivot workbooks from real-time business data for faster, more accurate insights

• Reduced complexity: Overcome cost and complexity of BI; shift IT resources from running ad-hoc reports to innovation initiatives

• Easy manageability: Custom code for management dashboard and scripted data source integration ease deployment and simplify administration

SKUs Components

BDA Server Dual Intel X5650 Processor with 96GB (1U)

Storage 8 x internal 300 GB SAS disks

Software Windows Server 2008 R2 EE, SQL Server 2008 R2 EE, SharePoint 2010 EE, PowerPivot

Infrastructure None (install in existing rack)

Services Software technical support

End-to-end, pre-configured stack quickly enables BI for Excel power users

Complete Data Warehouse Solution

Flexibility and Choice Massive Scalability at a Low Cost

Microsoft Data Warehouse VisionMake SQL Server the fastest and most affordable

database for customers of all sizes

Simplified Data Warehouse Management

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions,

it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.