building a dw - chapter 2

7/31/2019 Building a DW - Chapter 2

http://slidepdf.com/reader/full/building-a-dw-chapter-2 1/3

Data Warehouse Architecture

1. A data warehouse has two main architectures: the data flow architecture and the system architecture .

2. The data flow architecture is about how the data stores are arranged within a data

warehouse and how the data flows from the source systems to the users throughthese data stores.3. The system architecture is about the physical configuration of the servers, network,

software storage, and clients.There are three things that need to be known:

a. What the data flow architecture is b. What’s its purpose isc. What its components are

4. There are 4 main data flow architectures :a. single DDS

b. NDS + DDS

c. ODS + DDSd. Federated data warehouse5. A data store is one or more databases containing data warehouse data, arranged in

A particular format that is different from the format in the source systems andinvolved in the data warehouse processes.

6. Data warehouse data can be classified into three types:

a. A user facing data store is a data store that is available to end users and isqueried by the end users and end-user applications.

b. An internal data store is a data store that is used internally by datawarehouse components for the purpose of integrating, cleansing, logging

and preparing data and it’s not open for query by the end users and end-user applications.c. A hybrid data store is used for both internal data warehouse mechanisms

and for query by the end users and end-user applications.7. A master data store is a user-facing or hybrid data store containing a complete set

of data in a data warehouse, including all versions and all historical data.

Based on the data format, you classify data warehouse data stores into four types:a. A stage is an internal data store used for transforming and preparing the

data obtained from the source systems, before that data is loaded to other data stores in a data warehouse.

b. A normalized data store (NDS) is an internal master data store in the formof one or more normalized relational databases for the purpose of integrating data from various source systems captured in a stage, before thedata is loaded to a user-facing data store.

c. The operational data store (ODS) is a hybrid data store in the form of oneor more normalized relational databases, containing the transaction data andthe most recent version of master data, for the purpose of supportingoperational applications.



d. A dimensional data store (DDS) is a user-facing data store, in the form of one or more relational databases, where the data is arranged in dimensionalformat for the purpose of supporting analytical queries.

• A relational database is a database that consists of entity tables with parent-child

relationships between them.• A normalized database is a database with little or no data redundancy in third

normal form or higher.• A denormalized database is a database with some data redundancy that has not

gone through a normalization process.• A dimensional database is a denormalized database consisting of fact tables and

common dimension tables containing the measurements of business events,categorized by their dimensions.

Some applications require the data to be in the form of a multidimensional database (MDB)rather than a relational database. An MDB is a form of a database where the data is stored

in cells and the position of each cell is defined by a number of variables called dimensions. Each cell represents a business event and the value of the dimensions show when and where this event happened. MDB is populated from the DDS.

1. An ETL package consists of several ETL processes. An ETL process is a programthat is part of the ETL package that retrieves data from one of several sources and

populates one target table.2. An ETL process consists of several steps. A step is a component of an ETL process

that does a specific task.3. An example of a step is extracting particular data from a source data store or

performing certain data transformations.

4. The mechanism to log the result of each step of an ETL process is called ETL audit.5. Examples of results logged by ETL audits are how many records are transformed

or loaded in that step, the time the step started and finished and the step identifier so you can trace it down when debugging or for auditing purposes.

6. The description of each ETL process is stored in metadata. This includes:• the source it extracts the data from,• the target it loads the data into• the transformation applied • the parent process• the schedule each ETL process is supposed to run

7. In data warehousing, metadata is a data store containing the description of the structure, data, and processes within the data warehouse. This includes:

• the data definitions and mapping,• the data structure of each data store,• the data structure of the source systems,• the description of each ETL process• The description of the data quality rules• A log of all processes and activities in the data warehouse.



• Data quality processes are the activities and mechanism to make sure that data in the data warehouse is correct and complete. Data quality processesalso cover the mechanism to report the bad data and to correct it. A data

firewall is a program that checks whether the incoming data complies withdata quality rules.

•

A data quality rule is the criteria that verify the data from the source systems are within the expected range and in the correct format.

One of the first things you need to decide when building a data warehouse system is thedata flow architecture. This affects the project plan and costs. The data flow architecture isdesigned based on the data requirements from the applications, including the data qualityrequirements.

1. Data warehouse applications require data in different formats.2. These formats dictate the data stores you need to have.3. If the application requires dimensional format, then you need to build a DDS.

4. Once you determine the data stores you need to build, you can design the ETL to populate

building a dw - chapter 2

Documents