CS 345:Topics in Data Warehousing

Thursday, September 30, 2004

Review of Tuesday’s Class

• Roles of Information Technology– 1. Automate clerical work – 2. Decision support

• OLAP vs. OLTP– Different query characteristics– Different performance requirements– Different data modeling requirements– OLAP combines data from many sources

• High-level course outline– Logical Database Design– Query Processing– Physical Database Design– Data Mining

Outline of Today’s Class

• Data integration

• Basic OLAP queries– Data cubes– Slice and dice, drill down, roll up– MOLAP vs. ROLAP– SQL OLAP Extensions: ROLLUP, CUBE

• Star Schemas– Facts and Dimensions

Loading the Data Warehouse

Source Systems Data Staging Area Data Warehouse

(OLTP)

Data is periodically extracted

Data is cleansed and transformed

Users query the data warehouse

Terminology: ETL

• ETL = Extraction, Transformation, & Load• Extraction: Get the data out of the source

systems• Transformation: Convert the data into a useful

format for analysis• Load: Get the data into the data warehouse

(…and build indexes, materialized views, etc.)

• We will return to this topic in a couple weeks.

Data Integration is Hard

• Data warehouses combine data from multiple sources• Data must be translated into a consistent format• Data integration represents ~80% of effort for a typical

data warehouse project!• Some reasons why it’s hard:

– Metadata is often poor or non-existent– Data quality is often bad

• Missing or default values• Multiple spellings of the same thing

(Cal vs. UC Berkeley vs. University of California)

– Inconsistent semantics• What is an airline passenger?

Federated Databases

• An alternative to data warehouses• Data warehouse

– Create a copy of all the data – Execute queries against the copy

• Federated database – Pull data from source systems as needed to answer queries

• “lazy” vs. “eager” data integration

Data Warehouse Federated Database

Query

Answer

QueryExtraction

Rewritten Queries

Answer

SourceSystems

SourceSystems

WarehouseMediator

Warehouses vs. Federation

• Advantages of federated databases:– No redundant copying of data– Queries see “real-time” view of evolving data– More flexible security policy

• Disadvantages of federated databases:– Analysis queries place extra load on transactional systems– Query optimization is hard to do well– Historical data may not be available– Complex “wrappers” needed to mediate between analysis server

and source systems• Data warehouses are much more common in practice

– Better performance– Lower complexity– Slightly out-of-date data is acceptable

Two Approaches to Data Warehousing

• Data mart: like a data warehouse, but smaller and more focused

• Top-down approach– First build single unified data warehouse with all enterprise data– Then create data marts containing specialized subsets of the

data from the warehouse

• Bottom-up approach– First build a data mart to solve the most pressing problem– Then build another data mart, then another– Data warehouse = union of all data marts

• In practice, not much difference between the two• Our book advocates the bottom-up approach

Data Cube

• Axes of the cube represent attributes of the data records– Generally discrete-valued /

categorical– e.g. color, month, state– Called dimensions

• Cells hold aggregated measurements – e.g. total $ sales, number

of autos sold– Called facts

• Real data cubes have >> 3 dimensions Jul Aug Sep

CAORWA

Red

Blue

Gray

Auto Sales

Slicing and Dicing

Jul Aug SepCAORWA

Red

Blue

Gray

Red

Blue

Gray

Jul Aug SepCAORWA

Blue

Jul Aug SepCAORWA

Blue

Jul Aug SepTotal

Querying the Data Cube

• Cross-tabulation– “Cross-tab” for short– Report data grouped by 2

dimensions– Aggregate across other

dimensions– Include subtotals

• Operations on a cross-tab– Roll up (further aggregation)– Drill down (less aggregation)

CA OR WA Total

Jul 45 33 30 108

Aug 50 36 42 128

Sep 38 31 40 109

Total 133 100 112 345

Number of Autos Sold

Roll Up and Drill Down

CA OR WA Total

Jul 45 33 30 108

Aug 50 36 42 128

Sep 38 31 40 109

Total 133 100 112 345

Number of Autos Sold

CA OR WA Total

133 100 112 345

Number of Autos Sold

CA OR WA Total

Red 40 29 40 109

Blue 45 31 37 113

Gray 48 40 35 123

Total 133 100 112 345

Roll upby Month

Number of Autos Sold

Drill downby Color

“Standard” Data Cube Query

• Measurements– Which fact(s) should be reported?

• Filters– What slice(s) of the cube should be used?

• Grouping attributes– How finely should the cube be diced?– Each dimension is either:

• (a) A grouping attribute• (b) Aggregated over (“Rolled up” into a single total)

– n dimensions → 2n sets of grouping attributes– Aggregation = projection to a lower-dimensional

subspace

Full Data Cube with Subtotals

• Pre-computation of aggregates → fast answers to OLAP queries

• Ideally, pre-compute all 2n types of subtotals• Otherwise, perform aggregation as needed• Coarser-grained totals can be computed from

finer-grained totals– But not the other way around

Data Cube Lattice

Total

State Month Color

State, Month

State,Color

Month,Color

State, Month, Color

DrillDown

RollUp

MOLAP vs. ROLAP

• MOLAP = Multidimensional OLAP

• Store data cube as multidimensional array

• (Usually) pre-compute all aggregates

• Advantages:– Very efficient data access → fast answers

• Disadvantages:– Doesn’t scale to large numbers of dimensions– Requires special-purpose data store

Sparsity

• Imagine a data warehouse for Safeway.• Suppose dimensions are: Customer, Product, Store, Day• If there are 100,000 customers, 10,000 products, 1,000

stores, and 1,000 days…• …data cube has 1,000,000,000,000,000 cells!• Fortunately, most cells are empty.• A given store doesn’t sell every product on every day.• A given customer has never visited most of the stores.• A given customer has never purchased most products.• Multi-dimensional arrays are not an efficient way to store

sparse data.

MOLAP vs. ROLAP

• ROLAP = Relational OLAP• Store data cube in relational database• Express queries in SQL• Advantages:

– Scales well to high dimensionality– Scales well to large data sets– Sparsity is not a problem– Uses well-known, mature technology

• Disadvantages:– Query performance is slower than MOLAP– Need to construct explicit indexes

Creating a Cross-tab with SQL

SELECT state, month, SUM(quantity)FROM salesGROUP BY state, monthWHERE color = 'Red'

Grouping Attributes

Measurements

Filters

What about the totals?

• SQL aggregation query with GROUP BY does not produce subtotals, totals

• Our cross-tab report is incomplete.

CA OR WA Total

Jul 45 33 30 ?

Aug 50 36 42 ?

Sep 38 31 40 ?

Total ? ? ? ?

Number of Autos Sold

State Month SUMCA Jul 45CA Aug 50CA Sep 38OR Jul 33OR Aug 36OR Sep 31WA Jul 30WA Aug 42WA Sep 40

One solution: a big UNION ALL

SELECT state, month, SUM(quantity)FROM salesGROUP BY state, monthWHERE color = 'Red‘UNION ALLSELECT state, "ALL", SUM(quantity)FROM salesGROUP BY stateWHERE color = 'Red'UNION ALLSELECT "ALL", month, SUM(quantity)FROM salesGROUP BY monthWHERE color = 'Red‘UNION ALLSELECT "ALL", "ALL", SUM(quantity)FROM salesWHERE color = 'Red'

OriginalQuery

StateSubtotals

MonthSubtotals

OverallTotal

A better solution

• “UNION ALL” solution gets cumbersome with more than 2 grouping attributes

• n grouping attributes → 2n parts in the union• OLAP extensions added to SQL 99 are more

convenient– CUBE, ROLLUP

SELECT state, month, SUM(quantity)FROM salesGROUP BY CUBE(state, month)WHERE color = 'Red'

Results of the CUBE queryState MonthSUM(quantity)CA Jul 45CA Aug 50CA Sep 38CA NULL 133OR Jul 33OR Aug 36OR Sep 31OR NULL 100WA Jul 30WA Aug 42WA Sep 40WA NULL 112NULL Jul 108NULL Aug 128NULL Sep 109NULL NULL 345

Notice the use of NULL for totals

Subtotals at all levels

ROLLUP vs. CUBE

• CUBE computes entire lattice

• ROLLUP computes one path through lattice– Order of GROUP BY list matters– Groups by all prefixes of the GROUP BY list

GROUP BY ROLLUP(A,B,C)•A,B,C•(A,B) subtotals•(A) subtotals•Total

GROUP BY CUBE(A,B,C)•A,B,C•Subtotals for the following:(A,B), (A,C), (B,C), (A), (B), (C)•Total

ROLLUP example

Total

State Month Color

State, Month

State,Color

Month,Color

State, Month, Color

SELECT color, month, state, SUM(quantity)FROM salesGROUP BY ROLLUP(color,month,state)

Logical Database Design

• Logical database design is the topic for the next two weeks.

• Logical design vs. physical design:– Logical design = conceptual organization for the

database• Create an abstraction for a real-world process

– Physical design = how is the data stored• Select data structures (tables, indexes, materialized views)• Organize data structures on disk

• Three main goals for logical design:– Simplicity– Expressiveness– Performance

Goals for Logical Design

• Simplicity– Users should understand the design– Data model should match users’ conceptual model– Queries should be easy and intuitive to write

• Expressiveness– Include enough information to answer all important

queries– Include all relevant data (without irrelevant data)

• Performance– An efficient physical design should be possible

Another perspective…

• From a presentation at SIGMOD*:

• Three design rules:– Simplicity– Simplicity– Simplicity

*by Jeff Byard and Donovan Schneider, Red Brick Systems

Star Schema

Sales

Date

Product Store

Promotion

Fact table

Dimension tables

Dimension Tables

• Each one corresponds to a real-world object or concept.– Examples: Customer, Product, Date, Employee, Region, Store,

Promotion, Vendor, Partner, Account, Department

• Properties of dimension tables:– Contain many descriptive columns

• Dimension tables are wide (dozens of columns)

– Generally don’t have too many rows• At least in comparison to the fact tables• Usually < 1 million rows

– Contents are relatively static• Almost like a lookup table

• Uses of dimension tables– Filters are based on dimension attributes– Grouping columns are dimension attributes– Fact tables are referenced through dimensions

Fact Tables

• Each fact table contains measurements about a process of interest.

• Each fact row contains two things:– Numerical measure columns– Foreign keys to dimension tables– That’s all!

• Properties of fact tables:– Very big

• Often millions or billions of rows– Narrow

• Small number of columns– Changes often

• New events in the world → new rows in the fact table• Typically append-only

• Uses of fact tables:– Measurements are aggregated fact columns.

Comparing Facts and Dimensions

• Narrow• Big (many rows)• Numeric• Growing over time

• Wide• Small (few rows)• Descriptive• Static

Facts Dimensions

Facts contain numbers, dimensions contain labels

Four steps in dimensional modeling

1. Identify the process being modeled.

2. Determine the grain at which facts will be stored.

3. Choose the dimensions.

4. Identify the numeric measures for the facts.

Grain of a Fact Table

• Grain of a fact table = the meaning of one fact table row• Determines the maximum level of detail of the

warehouse• Example grain statements: (one fact row represents a…)

– Line item from a cash register receipt– Boarding pass to get on a flight– Daily snapshot of inventory level for a product in a warehouse– Sensor reading per minute for a sensor– Student enrolled in a course

• Finer-grained fact tables:– are more expressive– have more rows

• Trade-off between performance and expressiveness– Rule of thumb: Err in favor of expressiveness– Pre-computed aggregates can solve performance problems

Choosing Dimensions

• Determine a candidate key based on the grain statement.– Example: a student enrolled in a course– (Course, Student, Term) is a candidate key

• Add other relevant dimensions that are functionally determined by the candidate key.– For example, Instructor and Classroom

• Assuming each course has a single instructor!