data warehouse design & dimensional modeling
DESCRIPTION
At Code Mastery Boston Aaron Lowe, Principal Consultant at Magenic, talks Data Warehouse Design & Dimensional ModelingTRANSCRIPT
Data Warehouse Design & Dimensional Modeling Aaron Lowe
Principal Consultant
@Vendoran
@SQLFriends
Who am I » Aaron Lowe
» Husband
» Father of 5
» Principal Consultant at Magenic
» Working with SQL Server since 1998, version 6.5
» MCITP 2005 and 2008
» Co-organizer of SQLSaturday Chicago
» Masters in Information Systems Management
» www.aaronlowe.net / @Vendoran
» sqlfriends.org / @SQLFriends
Data, Data everywhere, but not a drop of Information
http://www.flickr.com/photos/walkingsf/5993167874/
The Data Person
http://www.flickr.com/photos/tantek/1360323838/
How can we get more out of our data?
http://www.flickr.com/photos/danahlongley/4472897115/
Leverage data to provide business insight
http://www.flickr.com/photos/juhansonin/4646203016/
Create a new Data Model in a Data Warehouse
Why a new Data Model?
What do we need?
Information – not just data » Collecting data
» Log Files » Clicks » How long? » How much?
» Prediction? » How Target Figured out Teen was pregnant -
http://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/
» The Numerati - http://www.amazon.com/The-Numerati-Stephen-Baker/dp/B003TO6G20/ - published 2009!
Relate data from multiple systems » The purpose of a data warehouse is to house standardized, structured, consistent,
integrated, correct, cleansed and timely data, extracted from various operational systems in an organization
» True picture of the business process
» Source Systems » Financial – AR/AP » Sales » CRM » HR » Application
Fast » It’s my information and I want it Now!
» Empower Users
» Exploratory
» Reads
» Large datasets
Why won’t existing models work?
What are they designed for? » Operational
» Preservation of data integrity
» Speed of recording of business transactions
» Often Many tables
» To free the collection of relations from undesirable insertion, update and deletion dependencies;
» To reduce the need for restructuring the collection of relations, as new types of data are introduced, and thus increase the life span of application programs;
» To make the relational model more informative to users;
» To make the collection of relations neutral to the query statistics, where these statistics are liable to change as time goes by.
» —E.F. Codd, "Further Normalization of the Data Base Relational Model"
Consistent » Partial data across
» Have the sale in the sales system » Represented in the inventory system » Don’t have the $ in the financial system yet
» Deleted on sources » Removed transactions » Archive » Legally destroy records can remove work product
» Incomplete data on source » Changes over time
Silo’d » How do we get the entire picture?
» Example:
» Cost of Sales? » Sales system – Sale Price
» Marketing System – $$ spent on Marketing
» Inventory System – $$ spent on inventory
» HR System – $$ spent on Employee
» IT Systems – $$ spent on Infrastructure
What will work?
http://www.flickr.com/photos/d-y-f/2870942257/
Designed for Users » De-normalized
» Fast Reads » Fast Reports » Limited JOINs
» Information » Scheduled » On Demand » Exploratory
» Information » Cross Functional » The more the better!
Inter-related data » Specifications for my Current Data Warehouse
http://www.flickr.com/photos/ross_goodman/3276964270/
Independent from Operational » Operational systems change
» Data will outlive Application
» Crashes
» Upgrades
» Breaking changes
» Single Source of truth
Logical Data Model
http://www.flickr.com/photos/doctorlizardo/6812846803/
Terminology
http://www.flickr.com/photos/doctorlizardo/6809564765/
Metadata Management » Business metadata
» What’s out there?
» Identify/Define
» Overloaded terms
» What is a customer?
» Process metadata
» DW process operations
» Asses system status
» Investigate problems
» Technical metadata
» Tables
» Fields
» Datatypes
Dimensions and Facts Dimensions Facts
Thing/Objects Measurements/Events
Nouns Verbs
Wide but short Skinny but long
Rows can exist independently Rows cannot exist independently
Descriptive Mostly Numeric and Additive
“By” words – FACT by Dimension
Quantity Ordered by Product by Customer by Date
Grain • Level of detail
• What is needed to meet business
requirements?
• What is possible to collect?
• How do you describe it?
• One row per X where X is the business
event
• One row per customer call
• One row per time sheet entry
• One row per employee status
change
• One row per order line item http://www.flickr.com/photos/frederikvanroest/3842334310/
Methodology
http://www.flickr.com/photos/doctorlizardo/6812847973/
Requirements – business focused » “Must embrace the goal of enhancing business value as the primary purpose.” –
Kimball
» “If your job is BI and you speak mostly to technical people all day, you are doing it wrong. Focus on first word - BUSINESS.” – Whitney Weaver (former Magenicon)
» Never ask “What do you want in the data warehouse?” Only one right answer - “Everything.”
» Ask questions that help you learn what the end user does
Kimball v. Inmon Ralph Kimball Bill Inmon
Kimballites Inmonites
Bottom Up Top Down
Dimensional Normalized
Star Schema 3rd Normal Form
Easier for the User More Difficult for the Users
Few JOINs Many JOINs
Dimension/Facts Entities
Complicated ETL Not as complicated ETL
Difficult to modify structure Easier to adapt
Not mutually Exclusive
Star vs. Snowflake Star Snowflake
ER resembles Star ER resembles Snowflake
Easier for the User More Difficult for the Users
Few JOINs Many JOINs
Faster Aggregations Slower Aggregations
Children with multiple parent tables
Normalized Dimensions
Snowflake is a variation on a Star, not an alternative http://www.flickr.com/photos/wandrus/6283157711/
History (ology?)
http://www.flickr.com/photos/doctorlizardo/6809564335/
Dimension Types » 0 – Inserts only, no updates or delete » 1 – Insert and updated to reflect current state » 2 – Slowly Changing Dimension (SCD)- multiple records to indicate different points in time
» 3 – multiple columns to indicate different point in time
» 4 – current value table and a history table » UNKNOWN values
Source Key Value StartDate EndDate
14 Blue 2012-01-01 2012-03-01
14 Green 2012-03-02
Source Key Value OldValue EffectiveDate
14 Green Blue 2012-03-02
Date and Time » Date
» Fundamental dimensions across all organizations and industries » Allows for trending across dates or periods » 1 row for every date in the years = 365 or 366 row/year » Use your words
» WeekDay » EndofMonth » Quarter » FiscalYear?
» Time » Not often needed, but becoming more popular » Allows for time based analysis for things like Status » 1 row for every time slice in a day – minutes? Seconds?
Surrogate Keys » New set of keys in the DW
» Protects against
» Source systems changes
» Single key for multiple source systems
» New rows that only exist in DW (UNKNOWN)
» Tracking over time (SCD)
Physical Data Model
http://www.flickr.com/photos/flying_cloud/2667218708/
Approach
http://www.flickr.com/photos/7506006@N07/7021456259/
Null – yay or nay » Same discussion as OLTP with a twist
» Purpose of DW is for reporting » Building on top of with :
» SSIS » SSAS
» Purpose of the Dimension UNKNOWN values
» Best practice is to avoid if you can, otherwise document » Some have separate values for UNKNOWN and NOT POPULATED » Default value instead
Aggregates » Minimize number of aggregates while maximize effectiveness
» Store or
» Can aggregate Facts
» Roll-up Dimension hierarchies?
» Can still be relational to other tables when necessary
Hierarchies » Example: » Date - Roll up by Month, Quarter or Year
» Variable depth – Self-referencing » Variable depth with historical – changing surrogate keys – ouch
» Track business process separately
Key Day Month Quarter Year
364 30 12 4 2011
365 31 12 4 2011
366 1 1 1 2012
367 2 1 1 2012
Size Matters
http://starwars.wikia.com/wiki/Rancor?image=Rancor-jpg
Data amount and size » Data Types?
» BLOB data?
» Identity columns (do you need bigint?)
» Data Profiling
» Collect source system sizes for data bringing over
» Add sizes of new row
» Don’t forget index size!!!
Partitioning » Usually lends naturally to partitioning large Fact tables by Date
» Larger Dimension tables can be partitioned as well
» Sometimes Old (SQL 2000) Partitioning is still better than SQL 2005+ partitioning
» Take ETL process into consideration
Archiving » Question: When is big too big?
» Answer: When performance impact outweighs need for data availability
» Many options:
» Backup to tape offline
» keep “Archived” DW available
» Records Retention – this could be your work product
Performance
http://www.flickr.com/photos/elfidomx/6026943114/
Hardware » Remember when the user said “It’s my data and I want it now”? » Buy
» Reference Architecture (Fast Track) » Appliances
» HP » Enterprise Data Warehouse » Business Decision » Business Data Warehouse » Enterprise Database Consolidation
» Dell » PDW
» Build » Reference Architecture (Fast Track) » SQLIO » Benchmark
Throughput » Amounts of data
» Not all of it will be in memory » Between ETL and reports, SP Cache might not be efficient » Need to tune those disks
» Reference Architecture(Fast Track) » Accepts that Procedure cache will stink due to data sizes » Instead small amount of RAM » Requires bandwidth of 400 GB/s per LUN
» Materialize data that makes reporting faster!! » More Denormalization » More Aggregations
» ReadOnly while not processing ETLs? (switch)
Parallelism » Multiple Data Files
» SQL writes proportional fill
» Multiple Filegroups » Partitioning scheme » Facts/Dimensions » Tables that are often joined » Big tables » NCIX vs. data
» Multiple LUNs » I am not a SAN admin nor play one on TV
» Normal SQL performance
Questions and Discussion time!