data warehousing basic concepts

8/3/2019 Data Warehousing Basic Concepts

http://slidepdf.com/reader/full/data-warehousing-basic-concepts 1/33

P e o p l e M a k i n g T e c h n o l o g y W o r k ™

DATA WAREHOUSING

Basics Concepts



Agenda

► Evolution of DWH► Why should we consider Data Warehousing solutions ?

► Definition of Data Warehouse

► Characteristics of DWH

► Difference between DW’s and OLTP

► DWH Life Cycle► DWH Architecture

► Dimensional Data Modeling

► Star Schema Design

► Fact Table

► Fact Granularity

► Dimension Tables

► Snowflake Schema Design

► Important aspects of Star Schema & Snow Flake Schema

► Data Acquisition (ETL)

► ETL Concepts



Evolution of DWH

Traditional approaches to computer system design during 1980’s

►Not optimized for analysis and reporting

►Company wide reporting couldn’t be supported from asingle system

►For developing reports often required writing specificcomputer programs which was slow and expensive



Why should we consider Data Warehousing solutions ?

When users are requesting access to a large amount of

historical information for reporting purposes, you should

strongly consider a warehouse or mart. The user will benefit

when the information is organized in an efficient manner for

this type of access.



Def . Data Warehousing

DWH is type of relational data base system specially

designed for query analysis processing rather than

transactional processing.

The DWH systems are also called as Historical Db’s,

Read only Db’s, Integrated Db’s, Decision Supporting

System, Executive info System, Business Info System.



►Subject Oriented

►Non Volatile

►Integrated

►Time Variant

Characteristics of DWH



Differences………………..

DWH database (OLAP) OLTP database

Designed for analysis of businessmeasures by category andattributes.

Designed for real time businessoperations.

Optimized for bulk loads and large,complex, unpredictable queriesthat access many rows pertable.

Optimized for a common set oftransactions, usually adding orretrieving a single row at a timeper table.

Loaded with consistent, valid data;

requires no real time validation.

Optimized for validation ofincoming data during

transactions; uses validationdata tables.

Supports few concurrent usersrelative to OLTP.

Supports thousands of concurrentusers.



OLAP Database (OLAP) OLTP Database

Multidimensional DatabaseStructures

Normalized DataStructures

Index - Many Index - Few

Joins - Few Joins - Many

Aggregated Data - More Aggregate Data - Few

No. of users - Few No. of users - More

Periodic update of data Data ModificationMore

Huge volumes of data Small volumes of data



DWH Life Cycle

Business Analyst

Data Modular

ETL Developer

Report Developer

Testing



DWH Architecture

Three common architectures are:

►DWH Architecture (Basic)

►DWH Architecture (With a staging area)

►DWH Architecture (With a staging area and data marts)



DWH Architecture (Basic)



DWH Architecture (with a staging area)



DWH Architecture

(with a staging area and data marts)



Dimensional Data Modeling

To develop a Star Schema design a Data Modeler followsdimensional modeling design aspect.

Dimensional modeling is a 3 stage process

►Conceptual modeling

►Logical Modeling

►Physical Modeling



Before start implementing the schema design aData modeler should understand the followingprocess

►Understand the clients Business requirements►Understand the grain of fact

►Designing of the Dimension tables

►Designing of the Fact tables



Example of Dimensional Data Model (Star Schema Design)



Fact Table

► Contain numeric measures of the business

► Contains facts and connected to dimensions

► two types of columns

facts or measures

foreign keys to dimension tables► May contain date-stamped data

► A fact table might contain either detail level facts or facts

that have been aggregated



Steps in designing Fact Table

► Identify a business process for analysis(like sales).

► Identify measures or facts (sales dollar).

► Identify dimensions for facts(product dimension, location dimension,

time dimension, organization dimension).

► List the columns that describe each dimension.(region name, branch

name, region name).► Determine the lowest level of summary in a fact table(sales dollar).



Types of Facts (Measures)

► Additive - Measures that can be added across all dimensions.

► Semi Additive - Measures that can be added across few

dimensions and not with others.

► Non Additive - Measures that cannot be added across all

dimensions.



In the example, sales fact table is connected to dimensions location, product, time

and organization. Measure "Sales Dollar" in sales fact table can be added

across all dimensions independently or in a combined manner which is

explained below.

► Sales Dollar value for a particular product

► Sales Dollar value for a product in a location

► Sales Dollar value for a product in a year within a location

► Sales Dollar value for a product in a year within a location sold or serviced by

an employee



Fact Granularity

► A fact table maintains a numerical info

► It is defined as the level at which fact info/- is stored.

► The level is determined by dimensional table.

Year?

Quarter?

Month?

Week?

Day?



Dimension Tables

► Contain textual information that represents attributes of the business

► Contain relatively static data► Are joined to fact table through a foreign key reference

► Are usually smaller than fact tables

Example of Location Dimension



Location Dimension

Location DimensionId

CountryName

StateName

CountyName

City Name Date Time Stamp

1 USA New York Shelby Manhattan 1/1/2005 11:23:31

AM

2 USA Florida Jefferson PanamaCity

1/1/2005 11:23:31AM

3 USA California Montgomery San Hose 1/1/2005 11:23:31AM

4 USA New Jersey Hudson Jersey City 1/1/2005 11:23:31AM

Location Dimension



Star Schema Design benefits

► Easy for users to understand

► Fast response to queries

► Support multi dimensional analysis

► Supported by many front end tools



Snowflake Schema Design

►Dimension table hierarchies are broken intosimpler tables

► In few organizations, they try to normalize the

dimension tables to save space

►Both Fact and Dimensional tables are Normalized

► Increases the number of joins and poor

performance in retrieval of data

►May become large and unmanageable

►Degrades query performance



Example of Snowflake Schema



Important aspects of Star Schema & Snow Flake Schema

► In a star schema every dimension will have a primarykey.

► In a star schema, a dimension table will not have anyparent table.

►Whereas in a snow flake schema, a dimension tablewill have one or more parent tables.

►Hierarchies for the dimensions are stored in thedimensional table itself in star schema.

►Whereas hierarchies are broken into separate tablesin snow flake schema. These hierarchies helps to drilldown the data from topmost hierarchies to thelowermost hierarchies.



Data Acquisition

► It is the process of extracting the relevantbusiness info/- from the different sourcesystems transforming the data from oneformat into an another format, integratingthe data in to homogeneous format and

loading the data in to a warehousedatabase.

►Data Extraction (E)

►Data Transformation (T)

►Data Loading (L)



Sample ETL Process Flow



ETL Process

The ETL Process having the following basic steps

► Is mapping the data between source systems and target database

► Is cleansing of source data in staging area

► Is transforming cleansed source data and then loading into the target

system



►Source System

A database, application, file, or other storage facility fromwhich the data in a data warehouse is derived.

►Mapping The definition of the relationship and data flow between

source and target objects.►Staging Area

A place where data is processed before entering thewarehouse.

►Cleansing The process of resolving inconsistencies and fixing theanomalies in source data, typically as part of the ETLprocess.



► Transformation

The process of manipulating data. Any manipulation beyondcopying is a transformation. Examples include cleansing,aggregating, and integrating data from multiple sources.

► Transportation The process of moving copied or transformed data from a

source to a data warehouse.► Target System

A database, application, file, or other storage facility to which the"transformed source data" is loaded in a data warehouse.



Thank You !!!

data warehousing basic concepts

Documents