Why Do EDWs Take So Long, Cost So Much and Can’t Answer My Questions?

DAMA Chicago| June 15th, 2016

2

SPEAKER

Charles LorenzSolution Architect, TimeXtender

3

What Business Says about Data Warehouses

Data warehouses, particularly EDW’s, are becoming increasingly necessary to compete in the marketplace.

It is agonizingly difficult to integrate all transaction system data, particularly if you have dozens of application systems.

It costs so much.

Developmental takes much longer than we were promised.

We can’t find people with the right skills and experience.

Data warehouses cannot respond to change quickly enough, particularly when we do more than one acquisition a year.

Data warehouses don’t live up to what we expected to have. When we need an enhancement, it usually takes a month or more to get anything done.

The designs are just too complex…

Nobody really understands what’s in the data warehouse and how to get information out.

4

EDWs Are So Large and Complex that:

*http://www.europe.gartner.com/3_consulting_services/cpm/critical_program_mgmt.jsp

** www.gartner.com/press_releases/asset_121817_11.html

66%of large-scale projects fail to achieve their stated business objectives, are delivered late, or are substantially over budget.*

G A R T N E R S A Y S

66%of DWs failed outright or had significant cost & time overruns or did not deliver the right functionality to support the business**

S T A N D I S H G R O U P S A Y S

5

What Organizations Want From a DW

or EDW

6

• A guarantee of success• Want it yesterday• Single homogenous data store• Answer any question across the

business• High quality design = Low TCO• 360 degree view • Respond quickly to changes &

acquisitions

7

Traditional Approach to Try to Provide This:

Hire or contract with a BI Team with track record of success

Buy prebuilt logical data model

o Complex

o Most 3NF LDMs - Have to convert to physical

Buy prebuilt physical database design for their industry

o EDW PDBDs few and far between

o There are several data mart point solutions

o Not scalable and performant

Data WarehouseAutomation (DWA)

9

And Data Warehouse Automation Is?

According to TDWI, DWA is:

“Using technology to gain efficiencies and improve effectiveness in data warehousing processes. Data warehouse automation is much more than simply automating the development process. It encompasses all of the core processes of data warehousing including design, development, testing, deployment, operations, impact analysis, and change management."

10

Conventional Wisdom

You can have it fast, cheap or good, but you can’t have all three

Fast + Cheap = Lesser Quality

Good + Fast = More Expensive

Cheap + Good = Slower Delivery

Cheap

Fast Good

11

DWA Turns Conventional Wisdom on its Head

A proper DWA platform can provide a data warehouse solution that is

Good: High quality architecture and components

Fast: Up to an order of magnitude faster delivery

Cheap: Very low cost compared to traditional methods

Cheap

Fast Good

Data Warehouse Automation

12

Major DW Cost & Schedule Drivers Data Architecture *

ETL development – up to 75% of development costs *

Testing *

Deployment *

Changes to DW database schema (Agile development)*

Changes to ETL to accommodate the above *

Refactoring (Agile development) *

Changes to ETL & analyses to accommodate the above *

Slowly changing dimensions *

Incremental loading *

Buggy code *

Documentation – which most often isn’t done*

Maintenance & evolution after initial implementation *

And many more….

13

Of Interest

ETL usually takes from 60 – 75% of budget & schedule

Post implementation costs can be 5 – 10 times implementation costs

Most DWs have little documentation

Turnover can drive up implementation & maintenance costs

New releases very costly

14

Real World Example

Customer had budget of $140K for:

Finance (GL,AP,AR,Budget,Forecasting)

Sales Operations & maintenance

Requested bids from severalDW consultancies

All considerably exceeded budget (by double )

Let’s take a look

15

Traditional Spend Rate By Phase with $140K Budget

Time

M A I N T E N A N C EB U D G E T

BUDGET ONLY LASTS FOR HALF OF ETL

Traditional

16

The Power of DWA with Same Budget (Spend Rate)

Time

M A I N T E N A N C EB U D G E T

Note: This includes cost of DWA software

Traditional DWAC

ost

17

Comparing Lifetime Total Cost of Ownership

Time

Co

st

DWA IMPLEMENTATION DONE

ESTIMATED TRADITIONAL IMPLEMENTATION DONE

Traditional DWA

18

DWA Benefits

Radically cuts time and cost

Helps make up for lack of expertise and experience

Much less complex

Get it done much faster and cheaper

Ideal for Agile approach

Low lifetime total cost of ownership

High quality architected solution

Up-to-date documentation generation – DD, mapping, lineage, etc.

DWA – A force multiplier!

Cheap

Fast Good

Data Warehouse Automation

Evaluating DWA Solutions

20

Evaluating DWA Solutions

Leading DWA software vendors

OPEN TOOLSAllows you to inspect and modify any element

Proprietary tools“Black Box” – complete dependency on the tool

http://eckerson.com/register?content=which-data-warehouse-automation-tool-is-right-for-you

21

Business Trends Affecting IT

Cloud transaction processing solutions

o Can be implemented quickly without IT involvement

Causes issues

o Wildly different data structures

o Many data concepts duplicated

o Virtually impossible to integrate without DW

o Many versions of the truth

o Chaotic situation

DWA is ideal to bring order out of chaos

DWA Demonstration

23

Agile Project

Source is AdventureWorks bicycle mfg & sales

Transaction system is very complex & confusing

Use agile DW approach

Epic is Sales business process

Sprint 1 will satisfy the Product Manager’s need to analyze product sales

23

24

Sales Subject Area (Mostly)

24

25

Sprint 1

25

26

Sprint 1 User Story

As Product Manager, I have an urgent need to analyze Product sales by:

Category, subcategory, model & name

Year

Quarter

Month

27

All of the Work Is Done in the Staging Area

STAGING AREA

STAGING AREA STORAGE

28

Denormalizing: Relating Tables Together

Can only copy data to/from tables that have a direct relationship

For tables with indirect relationships, must get needed data into a table that is directly related to the target table.

ProductCategory

Product

INDIRECT

DIRECT

DIRECT

DIRECT

ProductModel

ProductSubCategory

Let’s Get Started!

Call to Action