layered scalable architecture - bi302

BI302

The Layered Scalable Architecture for SAP NetWeaver BW on a Corporate Scale

Juergen Haupt Architect, SAP Intelligence Platform and NetWeaver RIG EMEA

Jens DoerpmundArchitect, SAP Intelligence Platform and NetWeaver RIG Americas

2009

© SAP AG 2009. All rights reserved. / TechEd 2009, BI302 /

Disclaimer

This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.


Agenda

1. EDW and the BW Layered Scalable Architecture (LSA) 2. BW Layered Scalable Architecture (LSA) –

The Reference Architecture for BW on Corporate/ Global Scale LSA Building Blocks LSA Data Layers & Data Domains

3. LSA Implementation – Unified Data Warehousing Transactional Data, Master Data

4. BW LSA Assistant Building Blocks - RDBMS & Columnar DMS

5. The Holistic View on BI & DWHing - Centralization & Federation

6. Summary


Correlation of BI Value & Data Logistics ExcellenceThe Need of Centralization

Operational Reporting

Spreadmarts Data MartsData

WarehousesEnterprise

DWArchitecture

Financial System

Executive System

Analytical System

Monitoring System

Strategic System

Business Service

Type of System

“ Drive theBusiness ”

“ Drive the Market ”

“ Monitor Performance ”

“ Empower Workers ”

“ Inform Executives ”

“ Cost Center ”

Executive Perception

Value

CostROI

Infant Child AdultTeenager Sage

Static Reports Spreadsheets OLAP/ Ad hoc Reports

Dashboards/ Scorecards

Analytical Tools

Predictive Analytics

Customer BI Embedded BI

Prenatal

ROI

Analytic Services

Operational Reporting

Spreadmarts Data MartsData

WarehousesEnterprise

DWArchitecture

Financial System

Executive System

Analytical System

Monitoring System

Strategic System

Business Service

Type of System






“ Cost Center ”







“ Cost Center ”


Infant Child AdultTeenager Sage

Static Reports Spreadsheets OLAP/ Ad hoc Reports

Dashboards/ Scorecards

Analytical Tools

Predictive

ROI

Beyond the Basics: Accelerating BI MaturityWayne W. EckersonDirector, TDWI ResearchThe Data Warehousing Institute 2007

Best practice: centralize before you federate‘...interestingly, organizations that try to federate development and administration before they have centralized these tasks often have limited success.’


Bill Inmon’s Corporate Information Factory & The Enterprise Data Warehouse (EDW)

Copyright ©1999 by William H. Inmon

General accepted: BI / Data Marts should be built on a consistent, solid data foundation –

The Enterprise Data Warehouse (EDW)


Bill Inmon’s Enterprise Data Warehouse (EDW) The Adult Stage

Copyright ©1999 by William H. Inmon

Enterprise Data Warehouse (EDW): A single instantiation of a data

warehouse layer for the entire corporation or big parts of the organization is often called the Enterprise Data Warehouse

EDW-Keywords offer a ‘single version of truth’ extract once & deploy many support the ‘unknown’ re-build new-build controlled redundancy provide a corporate memory

Conceptual Details Integrated historical complete comprehensive application neutral granular corporate owned non-volatile…


TDWI – Accelerating BI MaturityThe Chasm on The Way to The EDW Shore

Figure 1. The TDWI Maturity Model shows how many organizations evolve their BI solutions. The Gulf and Chasm represent places where many companies get stuck; the bell shaped curve represents the number of companies in each stage; and the labels above the bell curve represent the data structure dimension in the model.

Business Value Integration

Consolidation

3. Child 4. Teenager 5. Adult 6. Sage

GULF“ Management Reporting ”

”

“ Data Marts ”

“ Data Warehouses ”

“DW ”

“ BI Services ”

1. Prenatal 2. Infant

GULFCHASM

“ManagementReporting”

“Spreadmarts”

“DataMarts”

“DataWarehouses”

“EnterpriseDW”

“BI Services”

Moving towards a centralized EDW based architecture means we have to overcome profound challenges (chasm): Non-IT ones like politics, missing strategy and sponsorship and IT ones .....


The Chasm on the Way to the EDW Shore EDW + X : The Challenging X of Today’s EDWs

Broader scopeCross corporate BI/ reportingOften local BI/ standard-reporting

Support mission/ business critical Data Marts99.6% availability from year’s perspective8 am local time report availability - globally

Highly restrictive TCO & TCD expectations

EDW

Principles

&

Goals

+

Highly volatile environment Continuous roll out Continuous BI projects

X

Today’s BW EDWs are often expected being the foundation for BI on all organizational levels of a corporation. As a consequence

development, administration and operation conditions are highly challenging. The BW Layered Scalable Architecture shall help you crossing the resulting chasm :

EDW + X = BW + BW Layered Scalable Architecture (LSA)

Complex conditions24x7, globalHigh volumes (data, meta data, user)


Crossing the EDW Chasm:The BW Layered Scalable Architecture

SAP Customer Experiences & Feedback :

Nestlé

Kraft Foods

Arla Foods

Adidas

Edeka

Beiersdorf

HenkelJapan

TobaccoPhilips

Samsung

Novartis

Syngenta

BASF

LandHessen

Shell Downstream

Mc Kesson

Statoil

Best Practices &

EDW Principles

BW Layered Scalable Architecture (LSA) –The Reference Architecture for BW on a Corporate Scale

SAP Consu

lting & In

dustries

SAP Dev & RIG

LSA

Nike


Agenda






6. Summary


DEPLOYING WEBI ON SAP

With large scale BWs there is only one way to survive:standardize, standardize, standardize ...

The LSA is all about

What & Where

to Standardize


The Layered Scalable Architecture Goals -Standardization, Transparency, Efficiency

architecturednon-architectured

Non-Architectured

D a

t a

f l o

w

D a

t a

f l o

w

LSA Architectured

Large scale BW Data

Warehouses (EDWs) should

follow architecture

principles like we can observe

constructing large buildings:

standardized, scalable, no squiggles,

efficient usage of materials, earth quake

save.


The BW Layered Scalable Architecture (LSA) describes the design of

service-level oriented, scalable, best practice BW architectures founded on accepted EDW principles*.

The BW LSA serves as a reference architecture to design transparent, complete, comprehensive

customer DWH architectures (Customer LSA).

The Customer LSA describes corporate standards to build BI applications in a

performant, maintainable, flexible manner.

BW Layered Scalable Architecture (LSA)

* As introduced in Bill Inmon‘s Corporate Information Factory (CIF)


BI-Project-Design

BW LSA And Customer LSA

BW LSA: The Reference Architecture

BI-Project-Design

Customer LSA : Standards - Handbook

Step 1:

Design Customer LSA

Step 2:

ApplyCustomer LSA

BI Project Design

Step 3:

Perfect Customer LSA

Step 4:

Update Customer LSA


LSA & Large Scale BW ScenariosDifferent Customer Starting Points

BW1

BW2

BW3

BWn

C-BW

Consolidation BW

M-BWBW-Old

Migration BW

BW1

BW2

BWn

DWHn

I-BW+

Integration BW

P-BW

NewCorporate

SAPERP(s)

Primary BW

+


Large Scale BW Scenarios and Customer Focus

1. Consolidation BW – Replacing several BWs by a single BW

2. Migration BW – Redesigning/ Reengineering a BW

3. Primary BW – BW as primary source for all BI applications/ reporting and all organizations (BW EDW in parallel to ERP roll-out)

4. Integration BW – BW as integrated source for cross-organizational BI/ Reporting in addition to an existing BW/ DWH landscape

The scenarios may overlap or occur in combination of each other

The architecture and modeling emphasis of customers may differ considerably



LSABuilding Blocks

Reference

Describescore structures &

definitions

LSA Implementation

Reference

Describesdesign standards to build BI applications founded on building

blocks

LSAOperations Reference

DescribesSupport

Scenarios

Meta Data Management Naming Conventions Organization (InfoAreas)

Life Cycles Information Meta Object LSA

Administration Data Base Housekeeping Monitoring

Transport

Security

Data Staging/ Management for transactional & master data Persistent Objects Flows - scheduled/ recovery Transformation Programming (Abap) Organization (Process Ch.)

BW LSA

Landmark Building Blocks Layers Domains Data Model &

Data Integration Assistant Building Blocks Data Quality Landscape ETL Storage Ownership/ Organize Development concept


Agenda






6. Summary


LSA Building Blocks – The Architecture Cornerstones

The LSA Building Blocks

are the cornerstones of your future architecture thus having a decisive influence on the overall success of your future BW EDW

describe the general BW EDW layout independent from concrete BI applications and BI projects.

Landmark Building Blocks deal with architecture areas that need fundamental decisions before doing implementations – they play the same role like the supporting structures (pillars & ceilings) of buildings.

Assistant Building Blocks deal with architecture areas that are normally less prior with respect to the point in time you should decide and roll out corporate standards.

LSABuilding Blocks

Reference

Describescore structures &

definitions

Landmark Building BlocksLayersDomains Data Model &

Data Integration Assistant Building Blocks

Data QualityLandscapeETL Storage Ownership/ OrganizeDevelopment concept


LSA Landmark Building BlocksLayers & Domains

There are two areas to standardize the architecture of persistent data stores:

1. Structure the data stores in a flow from PSA to InfoCubes with respect to their role and the offered services – define Data Layers

2. Structure (split/ collect) the data within the layers to guarantee Layer and advanced services – define Data Domains

LSA ArchitecturedNon-Architectured

Domain

Layer

data

flo

w


LSA Landmark Building Blocks BW Data Model/ Data Integration

What has a SAP BW data model strategy to consider ?

BW has to cover reporting requirements from various organizational units:

Support of group and local business Data Marts (BW InfoProvider)

Support of group and local master data (BW InfoObjects)

SubOrg Data Marts

BW data model

Local master data Local Data Model: E.g. Compounded InfoObjects

GroupData Marts

Group master data Group Data Model


LSA Assistant Building Blocks I

ETL (Extraction, Transformation, Load)

Do you expect extensively data from non-SAP sources? If the answer is ‘yes’, it is meaningful investigating an ETL-tool like SAP Data IntegratorIf SAP systems are the initial and primary BW EDW sources, you just don’t care. May be later

Data Quality

Do you have considerable data quality issues?If the answer is ‘yes’, it makes sense thinking about a Data Quality tool. If integrated SAP systems are the initial and primary BW EDW sources, you don’t care.

Landscape often reduced to ‘Do I need a single or a multiple BW instance’ landscape. This topic has become more and more an assistant one, because of the arrival of new technologies and the transparent support by BW (BW Accelerator, Near Line Storage s. ‘Storage’). The landscape architecture comes into focus if we have to support mission critical BI or to observe legal restrictions or with other customer specific requirements if it comes to agile BI and local autonomy (federated landscapes, SAP Newton appliances and BW EDW)


LSA Assistant Building Blocks II

Storage Traditionally all data of a BW DWH are hosted by an RDBMS. This is for large scale BW EDWs not adequate (this applies also to smaller BWs) Use RDBMS compression if available and usable (e.g. DB2 ‘deep compression’)Rarely used data should be hosted by Near Line Storage (NLS). NLS tools compress your data up to 95% (e.g. SAND) without destroying your Service Level Agreements (SLAs). The BW Accelerator (BWA) must be part of the overall architecture

Ownership/ Organization

Designing, implementing and operating a BW EDW need strong governance.

A BI Competency Center (BICC) should be established if not existing yet.


Agenda






6. Summary


‘Layering’ means horizontal structuring/ modeling of BW The data flows are organized in a unified, service-oriented way. Parameters are: Coverage, comprehensiveness (Process, User demands) Granularity History (completeness) Reusability (BI application scalability) Recovery (robustness, availability) Quality Integration status Access-rights (End-user) Life-cycle .....

LSA Landmark Building Blocks Data Layer/ Layering of Data

Non-Architectured

D a

t a

f l o

w

D a

t a

f l o

w

Value of Data Layers:+Highly Transparent & Flexible

+Development, Maintenance+Administration, Operations+Database-Integration

LSA Architectured

Layer


LSA Reference Layers Quick Intro

LSA

Reporting Layer (Architected Data Marts)

Business Transformation LayerBusiness Transformation Layer

Operational D

ata Store

Operational D

ata Store

Data Propagation LayerData Propagation Layer

Quality & Harmonisation LayerQuality & Harmonisation Layer

Corporate MemoryCorporate Memory

Data Acquisition LayerData Acquisition Layer

Virtualization Layer

extractor inbox, 1:1 from extraction, temporary

source system like service level,comprehensive, complete, master the unknown, long term

create harmonized view, guarantee quality

EDW layersReusableCorporate owned Granular

BI Applications/Architected

Data Mart Layers

digestible, ready to consume, integrated, unified data

apply business logic

reporting, analysis ready abstraction near real time, operational like


LSA Reference Layers BasicsFrom Acquisition to Reporting Layer

SourceSystems

EDW Layers

Data flow

Corporate Memory

BusinessTransformation

Layer

Reporting Layer

Propagation Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

DM1

DM3

DM2Layering applies to transactional & master data!To master data however in a simpler form


LSA Reference Layers Acquisition Layer

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

DM1

DM3

DM2

Acquisition Layer Per DataSource per source system

DataSource should be comprehensive (offer all relevant process information)

Fast inbound & outbound to targets Accept data from extraction with as less

as possible overhead – no early checks, merges, transformations i.e. (1:1)

Stamps the extracted records uniquely for consistent internal DWH processing and tracking (request handling) Provides abstraction of DWH from sources Provides short term history of extracted data for immediate / short term data inspection


LSA Reference Layers Corporate Memory Layer

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

DM1

DM3

DM2

Corporate Memory (CM)‘Think about the Corporate Memory Layer as your DWH and BI life insurance‘:Mastering tasks, which are unforeseeable (‘the unknown’) and manageable only violating SLAs:

Avoiding re-init from source(s) (Disaster-) recovery Enhancing and new build of Data Marts Change of source master data transformations Reorganizations

the CM layer offers a source system like Service Level: Granular data Comprehensive data (high coverage of underlying process) Complete history of loaded data


LSA Reference LayersHarmonization & Quality Layer

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

DM1

DM3

DM2

Harmonization & Quality Layer The services of this layer are data and data model integration and quality checked data e.g. Technical harmonization

format, length simple content harmonization (e.g. date) integrating local master data to a single model (key compounding, concatenation)

Integration of local master data to group master data (group data model) Checking Integrity of loaded data Unification of data: adding e.g. load time, origin.. Advanced content cleansing

e.g. detect duplicates, address cleansing etc. -> Data Services

...Harmonizing data and guaranteeing quality of data can mean high efforts or no efforts at all. Customers who invested in process integration and common coded/ integrated master data will harvest now.


Detailing LSA Reference LayersPropagator Layer I

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

DM1

DM3

DM2

Propagator LayerThe Propagator Layer is the interface to all business related Layers. It stands for ‘extract once deploy many’ i.e. scalability of Data Mart development Propagator Layer Objects offer data that are easy to digest, ready to consume for all business purposes (Data Marts internal & external)

Propagation Layer


Propagation Layer

Detailing LSA Reference LayersPropagator Layer II

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

DM1

DM3

DM2

Easy to digest means standardized, unified data :From Harmonization & Quality Layer transformations we get:

Compliance of data to corporate quality and consistency standards Integration of local master data to group master data for group reportingIntegration of local master data into a single BW data model (compounding)

Easy to digest means standardized, unified data :Propagator Layer adds:

Option to enhance or merge Data to simplify Data Mart building - but no business specific transformations (no flavor for all EDW Layers)

Unified Data transfer behavior out of Propagator Objects (e.g. additive delta) Suitable portions of data (-> Domains) to fulfill SLAs related to local autonomy,

report availability, robustness, recovery, administration and operations


Detailing LSA Reference Layers Propagation Layer & Trimming Data

DataSource ASource 2


‘DataSource A’Propagator

3. Collect

DataSource BDataSource A

‘DataSource A & B’Propagator

2.Merge

DataSource A

‘DataSource A +’Propagator

1.ExtendAdd data

DataSource A

‘DataSource A’Propagator 1

4.Split


Propagator DSOs: Unified BI application experience – additive delta

∆ via

queue

∆via

generic

∆viafull

moving∆

via full

completefull

incompletefull ?

5.Unify


LSA Reference LayersData Mart Layers

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

DM1

DM3

DM2


Detailing LSA Reference LayersSub-Layers of Reporting/ Data Mart Layer

Access Abstraction-/Virtual Layer

Reporting/ Data Mart Layer

Analytics-/Dimensional Layer

Flexible Reporting/Granular Reporting

Layer

highly granular highly comprehensive short life cycle flat, multidimensional

less granular less comprehensive long life cycle multidimensional optimized performance

abstract from physics ‘virtual’ flexible ‘view’ generation protect front-end investments

Planning Layer

dedicated for planning direct input structures


From Reference LSA to Customer LSA Example: Customer Adaptation of LSA Layers

LSA


Business Transformation Layer

Op

eration

al Da

ta S

tore

Data Propagation Layer

Quality & Harmonisation Layer

Corporate

Memory

Data Acquisition Layer

Reference LSA

Customer LSA

AcquisitionLayer

CorporateMemory

Layer

PropagationLayer


Layer

FlexibleLayer

DimensionalLayer

VirtualLayer

(YADSSnnn)

YCDSSnnn

YPDSSnnn YBAPPnnn

YFAPPnnn

YVAPP1nn

YVAPP2nn

YDAPPnnn

D a t a f l o wlookup

1:1Unlink

UnflavoredIntegratedGranular

Ready to consume

Apply business-logic

ReportingGranular

ReportingPerformance

Long term

AbstractionFlexible

CompleteComprehensiveMost granular


LSA Layers @CustomerSummary

The LSA describes services that are relevant at large scale BW implementations. The services are grouped by Layers.

The LSA suggested Layers are a basic offering. Based on customer situation and preferences additional ones may be introduced even later on (Customer LSA).

As with all LSA areas customer adaptation of Layers follows the 80:20 rule i.e. concentrate on services first, which offer the most benefit for the majority of data

Not all Layers have necessarily own persistent InfoProviders for all data flows. Whether and when persistent InfoProviders exist are given in the LSA Implementation standards

The Customer LSA Layer definitions apply throughout the BW EDW – No exceptions without approval!

∑


Agenda






6. Summary


LSA Reference Layers Highly Simplified -Layering Does Not Resolve All Challenges

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

How to support: a world wide continuous roll-out BW-wide load scalability 24x7 operations & administration different reliability of sources local autonomy overall local & group SLAs...

?


EDW Life Cycle DimensionsIncremental Implementation & Scalability

2. Incremental in terms of organizational coverage – organizational roll-out

De

ma

nd

Pla

nn

ing

Ge

ne

ratin

g D

em

an

d

Pro

cure

2 P

ay

CO

PA

..... .....

EDW

nBI Application Coverage & EDW

Org

Un

it A

Org

Un

it B

Org

Un

it C

Org

Un

it N

..... .....

EDW

nOrganizational Coverage & EDW

Needs Scalability that is addressed by

(EDW)- layers

Needs Scalability that is addressed by

domains (& data model)

1. Incremental in terms of functional coverage – BI application roll-out

An EDW implementation is always an incremental one, never a big-bang


LSA Landmark Building Blocks Data Domains

Non-Architectured

D a

t a

f l o

w

D a

t a

f l o

w

LSA Architectured

Layer

Domain

Domains means structuring/ modeling of data within the layers: Transparent, disjunct structuring of transactional data using stable criterion.Target is the support of: Independency/ autonomy of organizations 24x7, time-zones Scalability / performance/ low latency

(parallel vers sequential) Challenging recovery-window Embedding into RDBMS Implementation & Operational robustness

Value of Data Domains:+Transparency & Flexibility

+Development, Maintenance+Administration, Operations

+Scalability & Robustness+Application+Load/ Query Performance+Database-Integration


LSA Data Domains BW Landscape Consolidation (Consolidation BW)

EuropeJapan

Asia Pacific

ERPERPERPERP

ERPERP

ERPERP

ERPERP

BWBWBWBW

BWBW

BWBW

BWBW ERPERP

North America

South America

A BW EDW replaces a bunch of existing BWs and/ or legacy DWHs (BI Consolidation) spread across the organization

To enable comparable services like we had in a distributed, multiple DWH instance world (yes, there are some nice things) we introduce Data Domains in a BW EDW that divide

the transactional data but use identical meta data & master data.

BWBW

Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world

X

X

Domain Americas

XX

Domain Europe

X

X

Domain Asia Pacific

BW EDW


LSA Data Domains BW in Parallel to ERP Rollout (Primary BW)

EuropeJapan

Asia Pacific

North America

South America

A single BW EDW shall offer standard reporting & analytics for all organizational units in a global ERP rollout.

To enable comparable services like we had in a distributed, multiple BW instance world we introduce Data Domains in a BW EDW that

divide the transactional data but use identical meta data & master data.


AMS Germany APAUS EMEA

Global ERP

BW EDW


LSA Data Domains Divide Data by Sources-‘Quality’ (Integration BW)

BW EDW

mainERP

mainERP

remoteERP 1

remoteERP 1

remoteERP 2

remoteERP 2

Main Domain Remote Domain

less stable, no controlstable, controlled



LSA Domain Concept - Strategic BW Partitioning Accross the Layers – For All Flows I

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

DM1

Domains apply to transactional dataNormally not to master data!

2LIS

_11_VA

ITM

Belongs to Domain A

Belongs to Domain B

Belongs to Domain C

Single E

RP


LSA Domain Concept - Strategic BW Partitioning Accross the Layers – For All Flows II

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer

Data Mart Layers

Harmonize/Quality

V- Layer

DM1

Domains apply to transactional dataNormally not to master data!

Belongs to Domain A

Belongs to Domain B

Belongs to Domain C

2LIS

_11_VA

ITM

Multiple E

RP


Transparent, Scalable Structuring of BW: LSA Domains & Layers

LSA





Corporate

Memory

Corporate

Memory


Access Abstraction Layer(MultiProvider)


Single Source System (Layer)Single Source System (Layer)

LSA





Corporate

Memory

Corporate

Memory




Multiple Source Systems (Layer)

Distribution actively designed:

Domains

Distribution inherited

LSA Domains distribute the transactional data for the entire BW EDW in a disjunctive manner. The meta data of domains are identical.

The LSA addresses an evolutionary EDW approach introducing Data Domains enabling ‘local’ BI services, 24x7 operations and administration without neglecting the broader EDW picture.


Criteria Defining LSA Domains

Golden rule:

As many domains as necessary – as less domains as possible

Rules of thumb:

Often a geography driven domain concept works well Often one basic domain per continent makes sense ( rough time zone handling) e.g. APA, EMEA, Americas time zone aspects may lead to e.g. 3 Asian domains East-Asia, Mid-Asia, West-Asia E.g. for continental BW EDW a starting point could be East, Middle, West...

Expected Volume (realistic volume estimates are a key input defining domains ) Large APA volume contribution & large (for your business important) countries may get an

own domain e.g. China and US

Independency (special service level agreement) for certain countries Important countries/ markets get an own domain

Robustness: take Quality/ stability of different sources into consideration (potential) instable data offerings from sources may lead to an own domain

Each customer may have additional criteria, normally it is a mixture of multiple items


LSA Building Blocks Define a Grid Throughout The DWH

Acquisition/PSA Layer

CorporateMemory

Layer

PropagationLayer


FlexibleLayer

DimensionalLayer

VirtualLayer

Lookup-tables

Asia

Europe

Americas

Germany

US

The LSA Building Blocks Grid stands for transparency throughout your BW. It allows a wide range of standardization and the formalization of standards for development, operations and administration


Agenda






6. Summary


LSA Building Blocks Define a Grid Throughout BW


CorporateMemory

Layer

PropagationLayer


FlexibleLayer

DimensionalLayer

VirtualLayer

Asia

Europe

Americas

Germany

US

EDW Layers Data Mart Layers


Domain US

LSA Building Blocks & Extract Once - Deploy Many

Propagators

Data Marts

DataSources

Ord

er

Del

iver

y

Sup

ply

Cha

in

Ord

er In

fo

1:n

Del

iver

y In

fo

Domain AMS

Ord

er

Del

iver

y

Sup

ply

Cha

in

Ord

er In

fo

Del

iver

y In

fo

Domain APJ

Ord

er

Del

iver

y

Sup

ply

Cha

in

Ord

er In

fo

Del

iver

y In

fo

Filling Domains with respect to domain criteria


LSA Basic Rules

Each InfoProvider belongs to exactly one Layer

The InfoProviders of the EDW layers are DataSource related

The InfoProviders of the Data Mart layers are BI application/ Data Mart Scenario related

InfoProviders of the Propagation layer are logical/ semantical partitioned with respect to strategic domains

InfoProviders of the Data Mart Scenarios inherit the logical/ semantical partitioning of the Propagators (except MultiProviders)

It follows that all InfoProviders of the Propagator Layer and Data Mart Layers belong exactly to one cell of the grid defined by the layers and domains

The Corporate Memory InfoProviders are not logical/ semantical partitioned with respect to domains

Harmonization layer InfoProviders (if any) are usually logical/ semantical partitioned with respect to domains

Rules of thumb!


LSA Grid & Naming Conventions IExample


CorporateMemory

Layer

PropagationLayer


FlexibleLayer

DimensionalLayer

VirtualLayer

YVAPP1XX

YVAPP1XX

Asia

Europe

Americas

Germany

US

YCdss100

YPdss1DX

YPdss1KX

YPdss1GX

YPdss1UX

YBapp1AX

YBapp1Dx

YBapp1Kx

YBapp1Gx

YBapp1Ux

YFapp1Ax

YFapp1Kx

YFapp1Gx

YFapp1Ux

YDapp1Ax

YDapp1Ux

YVapp1xx

YVapp1xx

YDapp1Gx

YDapp1Dx

YDapp1Kx

YPdss1Ax

YFapp1Dx


YPdss1Ax:Y : OwnerP : LAYERdss : DataSource/ app: Application1 : Sequence-noA : DOMAINx : Further Partitioning


LSA Grid & Naming Conventions II

LSA Naming Conventions for InfoProviders

The LSA grid defined by Layers & Domains suggests for InfoProvider a naming based on 8 bytes: Layer: 1 byte/ letter Domain: 1 byte/ letter Additional logical/ semantical partitioning placeholder: 1 Byte Area: 2 to 3 bytes

– A DataSource related identifier for all EDW layers related InfoProvider– A Business scenario related identifier for all Data Mart Layers related InfoProvider

Sequence number: 1 byte if more than one InfoProvider per cell of the grid

Remaining bytes for customer specific requirements e.g. ownership

Example:

1-Y : Corporate is owner 2-P : Propagator Layer InfoProvider 3-5 SHA : Area filled from Sales Order Header DataSource 6-U : Domain US 7-1 : 1st InfoProvider 8-X : no further logical/ semantical partitioning

The LSA naming conventions allow easy introduction of additional layers and domains if required

The LSA naming conventions allow easy introduction of additional layers and domains if required

Y P SHA U 1 X

1 2 3-5 6 7 8


LSA Implementation Overview Transactional Data Processing

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer

Data Mart LayersHarmonize/

QualityV-

Layer

InfoCube/ BWA Index Logical (semantical) partitioned - Domains

MultiProvider

Write optimized DSO Branched out from daily flow Not Logical (semantical) partitioned – No Domains applied

Normal DSO Business key Logical (semantical) partitioned – Domains applied

InfoSources shield layer InfoProviders

PSA Elements

Domains apply on Top of PSA via logical (semantical) partitioning

Customer transformationsLSA Unification Data Data behavior

2L

IS_

11_

VA

ITM

2L

IS_

11_

VA

ITM


LSA Implementation Overview Role of InfoSources

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer


QualityV-

Layer


MultiProvider




PSA Elements



2L

IS_

11_

VA

ITM

2L

IS_

11_

VA

ITM


LSA Layer ImplementationShield Layers with InfoSources I

Layer in focus DSO

Inbound InfoSource Shield of Layer in focus-

unified view of layer as target

Outbound InfoSource Shield of Layer in focus –

unified view of layer as source

Central Transformation from Previous Layer to Layer in

Focus

Outbound InfoSource Shield of Previous Layer-

unified view of layer as source

Inbound InfoSource Shield of Subsequent Layer- unified view of layer as target

Central Transformation from Layer in Focus to Subsequent

Layer

No Transformation

No Transformation


LSA ImplementationShield Layers with InfoSources II

Central Transformation from Layer in Focus to Subsequent

Layer :No change

Central Transformation from Previous Layer to Layer in

Focus:No change

No Transformations

No Transformations

Layer in focus: Two new DSOs (logical Partitions)

of new Domains introduced


LSA ImplementationShield Layers with InfoSources III

InfoSources shield persistent InfoProviders of the Layer

Persistent InfoProviders of layers should be shielded by InfoSources (inbound and outbound InfoSources).This guarantees Flexibility (creating logical/ semantical partitions for a new domain) Transparency (transformation rules always at defined locations)


LSA Implementation Overview Building Domains

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer


QualityV-

Layer



PSA Elements


2L

IS_

11_

VA

ITM

2L

IS_

11_

VA

ITM


LSA Domain ImplementationLogical/ Semantical Partitioning

LSA Domain Implementation - Basics

1. LSA Domains define a strategic disjunctive distribution of transactional data within the Layers throughout the entire BW

2. LSA Domains are equal with respect to meta data.

3. Domains should apply to all data flows on top of the Acquisition Layer/ PSA (go for the earliest reasonable and possible point in time)

4. Applying Domains to data of a DataSource may mean that extracted data are either split or collected

5. Domains do not apply to the Corporate Memory Layer and Master Data

6. LSA Data Domains for an InfoProvider are implemented using the BW best practice ‘Logical Partitioning’. With BW 7.20 there will be the functionality ‘Semantical Partitioning’, which automates the replication process .

7. A BW EDW roll-out by Domains is automated by the new BW 7.20 ’Copy Data Flow‘ Wizard


LSA Domain ImplementationLogical/ Semantical Partitioning


CorporateMemory

Layer

PropagationLayer


Layer

FlexibleLayer

DimensionalLayer

VirtualLayer

(YADSS100)

YCDSS100

YBAPP1AX

YFAPP1AXYDAPP1AX

YVAPP1XX

YVAPP1XX

Asia

YPDSS1AX

YBAPP1GXYDAPP1GX

Americas

YPDSS1GX

Logical/Semantical Partitioning ____


LSA Domain ImplementationNew SPO (Semantical Partitioned Object) I

Source 1 Source 2

Have to exist

InfoSource

APA EMEA

InfoSource

Source

InfoSource

Source

AMERICAS

InfoSource

Partition 1 Partition 2 Partition 3

Generated by SPO

Overflow Partition

InfoSource

Master Transformation

Reference Structure

Simple (and identical) 1:1 Transformations

Acquisition Layer

Subsequent Layers

DomainsNW BW 7.20


LSA Domain ImplementationNew SPO (Semantical Partitioned Object) II

With BW 7.20 logical partitioning will become a feature called ‘semantical partitioning’

InfoCubes and DSOs may be defined as ‚Semantical Partitioned Object‘ – SPO

SPOs support actively the generation of all necessary meta data (DSOs, In & Outbound InfoSources, Transformation Rules, DTPs, Process Chains...)

NW BW 7.20


LSA Domain ImplementationRoll-Out by Domain – Copy Data Flow


CorporateMemory

Layer

PropagationLayer


Layer

FlexibleLayer

DimensionalLayer

VirtualLayer

(YADSS100)

YCDSS100

YBAPP1AX

YFAPP1AXYDAPP1AX

YVAPP1XX

YVAPP1XX

Asia

YPDSS1AX

YBAPP1GXYDAPP1GX

Americas

YPDSS1GX

New BW development feature:Copy Data Flow ____

NW BW 7.20


LSA Domain Implementation Roll-Out by Domain - Scenario

Frequently population of the domains is done one by one instead of all at the same point in time.

Scenario: Global/ central BW for 3 different regionsDomains defined by regions

Asia Pacific and Japan (APJ) Europe, the Middle East and Africa (EMEA) North, Central and South America (NCSA)

Go life region by region One productive ERP for each region is connected to the productive BW

ModelingOnly one development ERP is connected to the development BW

Use a source-system dummy in your development BW for each productive ERP (new NW BW 7.20)

Build data flows and process chains for first domain (region). This data flows and process chains are copied with new BW NW 7.20 feature ‘Copy Data Flow’ to the next domain before this region goes life. The existing data flows and process chains serve thus as a template (low TCD)


LSA Domain ImplementationRoll-Out by Domain – Copy Data Flow Wizard

Data Flow Copy Wizard

BW 7.20

Note: This example illustrates only the Copy Data Flow Wizard. From LSA point of view this example is not ideal as no inbound and outbound InfoSources are used!


LSA Implementation Source System Dummies

Maintain your source system dummies in the Data Warehousing Workbench (RSA1) Only done once for each productive source system connected to your productive BW Mapping to a productive source system is maintained in your productive BW Example: source system dummies refer to the BR9CLNT000 and name them

ERP Asia Pacific and Japan (ERPAPJ) ERP Europe, the Middle East and Africa (ERPEMEA) ERP North, Central and South America (ERPNCSA)


LSA Implementation Overview LSA Data Unification

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer


QualityV-

Layer


MultiProvider




PSA Elements



2L

IS_

11_

VA

ITM

2L

IS_

11_

VA

ITM


LSA ImplementationAcquisition Layer Outbound

?

?

?

?

? Customer specific data transformations

Master data integration, quality of data, referential integrity etc

LSA promotes data unification

Managing domains and other administrational tasks need additional standard information (InfoObjects), which are stored in all InfoProviders of the EDW Layers

LSA promotes data behavior unification

Propagators should offer data to data marts, which behave as much as possible in the same way (additive delta)


LSA Implementation LSA Data Unification & Domain Characteristics

BW EDWAll data

Domain AAPA

Domain BEMEA

Country/ Market

E.g. UK

Org-unit from source

e.g. 0COMPCODE

=UK50

Org-unit from source

e.g. 0COMPCODE

=UK87

Country/ Markete.g. DE

Domain CAMS

The LSA promotes that all loaded records will be ‘stamped’ assigning a stable ‘domain- driving’ characteristic like market/ country to organizational criteria like in this example 0COMPCODE coming from source system.

The assignment is maintained in a control table.Simple example:

YFIELD YIOBJECT YVAL YDOMAIN YBWORG

BUKRS 0COMPCODE DE99 B DE

BUKRS 0COMPCODE UK50 B UK

BUKRS 0COMPCODE UK87 B UK

BUKRS 0COMPCODE US20 C US

YBWORG

YIOBJECT

YDOMAIN

assign

assign


LSA ImplementationLSA Promoted Data Unification

The PSA handling standardizes the loaded data for consistent internal BW processing stamping each record with request-no, package-no and record-no

Processing data from PSA to any target the LSA suggests adding useful information (InfoObjects) for domain management and other administrational tasks:

1.YDOMAINDomain criteria: contains the value of the logical/ semantical partition a record belongs

to

2.YBWORG BW organizational criteria: contains the value of the organizational unit a record

belongs to from business perspective. It is a sum of source-system organizational units

3.0SOURSYSTEMOrigin

4.YLTIMELoad timestamp

5.Other customer defined useful information e.g. Request-ID Any (stable) criteria that increases transparency


APA EMEA Americas

1. Unification transformations Add Domain Characteristics

YDOMAINYBWORG

Add Admin Characteristics0SOURSYSTEMYLTIME

Use start routine2. Simple data model and value

(compounding) harmonization

Propagator InfoSource Propagators :

no transformations

Flow Logic:

CM VAITM

LSA ImplementationLSA Data Unification & Data Transformation

Flow ControlTable:

YFLOWCTRL

Propagator InboundUnification InfoSource Propagators

InfoSource: additional transformations (e.g. value mapping)

depending on status of loaded records

Unification InfoSource CM :

no transformations

PSA PSA

ERP B

2LIS_11_VAITM

ERP A

2LIS_11_VAITM

Acquisition Outbound

LSA Unified Data


LSA Implementation & Corporate ERPSimplified Harmonization & Quality Layer

PSA

2LIS_11_VAITM

Propagator Inbound

P-BW

NewCorporate

SAPERP(s)

Primary BW

+

Corporate ERP offers integrated data

i.e. group equal local master data high quality data

A pragmatic, simplified processing is possible

LSA Unified Data CM VAITM


LSA Implementation Overview Filling Domains

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer


QualityV-

Layer



PSA Elements


2L

IS_

11_

VA

ITM

2L

IS_

11_

VA

ITM


LSA ImplementationFilling Domains – Split and/ or Collect

Filling the logical/ semantical partitions (defined by the strategic domains) from PSA data may mean

1. The flow to the subsequent layers is split into different ones or

2. The data of multiple PSAs of the same DataSource but different source systems are collected or

3. Both splitting and collecting i.e. flows from one PSA element are split whereas others are collected (not in picture)


LSA ImplementationFilling Domains – Split Techniques

DTPs

Splitting data of a PSA element into several logical/ semantical partitions of a Propagator DSO can be done directly on top of the PSA element or on top of an intermediate so called ‚Pass Thru‘ DSO (write optimized)

PSA

The early PSA-based split

APA EMEA Americas

The ‘Pass Thru’ DSO (Write Optimized) based split

PSA

Pass ThruWO-DSO

APA EMEA Americas


LSA ImplementationFilling Domains – PSA based Split

The earlier the data are split into logical partitioned staging flows the better for performance

The earliest time to split data in BW is the extraction (DTP) from the PSA to the next stage (slim staging)

PSA-based splitting is in most cases the preferable modelling option

PSA

The early PSA-based split

APA EMEA Americas


LSA ImplementationFilling Domains – PSA based Split & DTPs

APA EMEA Americas

PSADataSource

Flow Logic:

Data Transfer:

Unification IS

In DTP-exit routine:Access Flow Control table:

DataSource, source system and the target of the DTP (note 1172175)

Return from Flow Control table:values (YVAL) of the partitioning field (YFIELD) .These values define the selection criteria of the DTP.

CM WO-DSO

Same DS diffSource

Flow ControlTable:

YFLOWCTRL

Propagator Inbound


LSA ImplementationFilling Domains – Controlling Filtering DTPs

Access Flow Control Table´:

YDS YFIELD YIOBJECT YVAL YDOMAIN YBWORG YSRC YTARGET

2LIS_11_VASCL BUKRS 0COMPCODE FR50 F FR S1 YPVAS1FX

2LIS_11_VASCL BUKRS 0COMPCODE US20 U US S1 YPVAS1UX

2LIS_11_VASCL BUKRS 0COMPCODE MX30 A MX S1 YPVAS1AX

2LIS_11_VASCL BUKRS 0COMPCODE MX40 A MX S1 YPVAS1AX


LSA Implementation Filling Domains – Pass Thru based Split

Pass Thru based split may be considered:If consistency checks are performed with error handling (DTP error-stack)

Error handling may become complex with PSA based split as each DTP has its own error-stack i.e. number or error-stacks equals number of domains for each flow

In this case the Pass Thru DSO is strictly an Object of the Harmonization & Quality Layer

General Aspects of Pass Thru DSOs:A dedicated Pass Thru DSO is preferred by some customers because of the behavior of the PSA and the more transparent (flow-) handling

Data have to deleted regularly (NW BW 7.01: Disable delta consistency check for write-optimized DataStore Objects)

For each DataSource a dedicated Pass Thru DSO has to be built

Additional persistency (performance, housekeeping)

APA EMEA Americas

PSA

Pass ThruWO-DSO


LSA Implementation Pass Thru based Split & Consistency Checks

APA EMEA Americas

PSA

DataSource

Flow Logic:

Consistency/ reference Checks (master data) may be designed early (2) (or after Pass Thru (3))

unification of data, data model and keys (compounding/ concatenation)

(1) CM is a pure copy of PSA (4) CM get‘s only checked data

CM WO-DSO

Same DS diffSource

Reference data

Pass ThruWO-DSO

Content of Pass Thru should be regularly deleted like PSA Propagator Inbound


APA EMEA Americas

PSA

DTPs:

CM WO-DSO

Same DS diffSource

LSA Implementation Pass Thru Dataflow & DTPs

Pass ThruWO-DSO

Flow Control Table not used in DTPs only in Transformation Single delta DTP from PSA to Pass Thru Single Error DTP

Single Error-DTP

DTPs to domain DSOs (Propagators) simple extractions using YDOMAIN (control table)


LSA Implementation Overview LSA Data Unification – Trimming Data

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

Reporting Layer

Propagation Layer

AcquisitionLayer


QualityV-

Layer


PSA Elements


2L

IS_

11_

VA

ITM

2L

IS_

11_

VA

ITM


LSA ImplementationPropagation Layer & Trimming Data




3. Collect



2.Merge

DataSource A


1.ExtendAdd data

DataSource A


4.Split



∆ via

queue

∆via

generic

∆viafull

moving∆

via full

completefull

incompletefull ?

5.Unify


LSA ImplementationData Propagator & Digestable Data - Example

Merge (Abap: PROVIDE)

Infotyp 0001 Infotyp 0002 Infotyp 0004

from to OrgUnit

01.01.09 31.12.09 50000001

01.01.10 31.12.99 50000002

from to Name

01.01.09 30.09.09 Müller

01.10.09 31.12.99 Meier

from to Handicap

12.02.10 31.12.99 50%

01.01.09 30.09.09 50000001 Müller -Propagator Content:

01.10.09 31.12.09 50000001 Meier -

01.01.10 11.02.10 50000002 Meier -

12.02.10 31.12.99 50000002 Meier 50%

Note: This is only an example: 0EMPLOYEE_ATTR DataSource merges InfoTypes: 0,1,7,8 all additional InfoTypes like 4 must be merged in the Propagator Layer

Merge HR-InfoType DataSources

© SAP AG 2009. All rights reserved. / TechEd 2009, BI302 / SRC3SRC2SRC1

DS DS DS DS DS DS DS DS DS DS DS DS DS DS DS DS

0EMPLOYEE_ATTR

0PERSON_ATTR

0HRPOSITION_ATTR

PA-Info-types

OM-Info-types

Customer Infot-types

DS

0EMPLOYEE_ATTR

0PERSON_ATTR

0HRPOSITION_ATTR

DS

0EMPLOYEE_ATTR

0PERSON_ATTR

0HRPOSITION_ATTR

DSO DSO DSO DSO DSODSO

DS DS DS DS DS DS DS DS DS

0EMPLOYEE_ATTR 0PERSON_ATTR 0HRPOSITION_ATTR PA-Infotypen OM-Infotypen Kundeneigene Infotypen

PA-Info-types

OM-Info-types


PA-Info-types

OM-Info-types


DSODSO DSO

ZEMPLOYEE

ZEMPLOYEE ZPERSON ZHRPOSITION

Data Acquisition Layer

Corporate Memory

Data Propagation Layer

Reporting Layer

Harmonization Layer

PROVIDE + gather

PROVIDE + gather

PROVIDE + gather

PSA PSA PSA PSA PSA PSA PSA PSA PSA PSA PSA PSAPSA PSA PSA PSA PSA PSAPSA PSA PSA PSA PSA PSAPSA PSA PSA

InfoSource

DSO

DSO

DSO

ZEMPLOYEE

ZPERSON

ZHRPOSITION

DSODSO DSO

ZPERSON ZHRPOSITION

LSA Implementation Trimming Data: Merge DataSources (Example)

Merged Propagators

CM: Merged Transformed Data

CM: extracted data 1:1


LSA ImplementationPropagation Layer & Trimming Data




3. Collect



2.Merge

DataSource A


1.ExtendAdd data

DataSource A


4.Split



∆ via

queue

∆via

generic

∆viafull

moving∆

via full

completefull

incompletefull ?

5.Unify


LSA ImplementationPropagator Unification

PSA

A-table Ch-Log

Acti-Queue

1. Transfer data to Propagator (1) DSO with added load timestamp and activate Propagator

2. Via generic extractor read all records where load timestamp is older than last load timestamp and load to Propagator DSO (2) • set all key-figures of selected records to zero

3. Activate Propagator DSO (3)• Propagator offers now complete additive delta

4. Transfer to application Layer (InfoCubes) (4)

Situation: Extractor offers ‘Incomplete Full’ Extractor offers full loads. A DSO cannot calculate a delta with respect to last load as the new full load does not contain records, which were deleted in the source or contain zero bookings


Agenda






6. Summary


LSA & Master DataSituation

Master data have two main tasks to fulfill:– Being target of lookups during transactional data load (referential integrity, adding

information– Being the shared dimensions of InfoCubes for reporting (MultiProvider

Today InfoObject hosted master data (P,Q,S,X,Y tables) serve for both purposes.

InfoObjectMaster data

Shared Dimension(Navigational Attributes)

Integrity, Lookups

Transactional loads


Issues with Data Mart Level Managed Master Data

Process C

hainD

ata Mart A

Transactional loads

With non-architected BWs the master data loads are often managed on BI application/ Data Mart level what leads to redundant, uncontrolled master data processing.This is unacceptable for large scale BWs

Data Mart level managed master data:

Master data Load e.g.

0CUST_SALES + Change Run

3:00

Process ChainMaster Data



Transactional loads

First Step: BW level managed master data:

Master data should be collected/ prioritized and loaded across the Data Marts thus become BW managed

Transactional loads



Process C

hainD

ata Mart B

1:00 1:15

Transactional loads

Process ChainData Mart B

Process ChainData Mart A

2:301:00 1:15


LSA Implementation Layering Master Data

LSA


Propagation/CM LayerPropagation/CM Layer



InfoObject tables

Master Data DSO

Shared Dimension x,y,p,q,s tables

(Navigational Attributes)

Integrity, Lookups

Picture shows master data with delta load, master data withfull loads need additional ‘assistant DSO’ to determine delta

Large scale BWs should layer the master data:

1.Separation of staging and reporting tasks InfoObject tables for

reporting Propagator Master Data

DSOs for staging 2.Storing master data history

(introducing ‘active from’ ‘active-to’ time-slice in Propagator Master Data DSO)


LSA Implementation Decoupling EDW & Data Mart Layer Loads

Process ChainsMaster Data

InfoObject Change RunPropagator/

Corporate MemProcess ChainData Mart A

LSA managed master data loads

EDW transactional & master data loads are decoupled from Data Mart & InfoObject loads

It still remains challenging at what point in time to schedule master data loads in a global system

Process ChainsEDW Master Data

Propagator for0CUST_SALES

Process ChainsEDW Transactional

Data

Data Mart LayerB Trans Layer

InfoObject Update




multiple loads

per day

loads by

business

requirements


LSA Implementation Real Time Data Acquisition (RDA) for Master Data

Process ChainsMaster Data

InfoObject Change Run

Propagator/Corporate Mem

Process ChainData Mart A

LSA managed master data loads (RDA)

With NW BW 7.20 there will be Real Time Data Acquisition (RDA) for Master Data .

This will increase the robustness of global BW EDWs considerably

RDAEDW Master Data

Propagator for0CUST_SALES

Process ChainsEDW Transactional Data


InfoObject Update




multiple loads

per day loads by

business

requirements

continuous loads

NW BW 7.20


SAP BW @Global CP CustomerLayered, Scalable Architecture & Business Value

Domains/ Logical BW Partitions:

Market reporting

BW MultiProvider: cross market reporting

EDW/ Corporate Memory: disjoint Granular- Most atomic Historical complete Comprehensive covers 95% of business in total ~ 150 TB allocated

1 : 1

Global EDW/ Corporate Memory in total ~50 TB

Global reporting/ dash boarding on integrated data with drill thru

to most granular level


Agenda






6. Summary


SAP BW EDW & RDBMS

BW 60 TB

Proof of Concept

onDB2

RDBMS & Space

- Compression


SAP BW LSA / RDBMS / Hosts Transparency of LSA Enables Embedding

SAP BW LSA Architecture

0

Dim

en

sio

n

Tab

les

Bas

is

Tab

les

Mas

ter

Dat

a

6 7 8 9 10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

1 3 4 52

MidSize

Objects (Fact)

8 Partitio

ns

DB2 Partitioning Layout

LPAR 0 - 0 sysXdb00pDB2 Partition 0

LPAR 1 - 0sysXdb10pDB2 Partition 6 … 13

LPAR 2 – 0sysXdb20pDB2 Partition 14 … 21

LPAR 3- 0 sysXdb30pDB2 Partition 12 … 29

LPAR 4 - 0 sysXdb40pDB2 Partition 30 … 37

LPAR 0 - 1 sysXdb01pStorage Agent





4 FC(tape)

8 FC(tape)

8 FC(tape)

8 FC(tape)

8 FC(tape)

4 FC(tape)

8 FC(tape)

8 FC(tape)

8 FC(tape)

8 FC(tape)

4 FC(FC disk)

8 FC(FC disk)

8 FC(FC disk)

8 FC(FC disk)

8 FC(FC disk)

LPAR 0 - 2 sysXdb02pApp Server

LPAR 1 - 2sysXdb12pApp Server




4 FC(FC disk)

8 FC(FC disk)

8 FC(FC disk)

8 FC(FC disk)

8 FC(FC disk)

pSeries Architecture

BW-LSA-Cell BW-DataClass(es)� Table space(s) DB2 Partition(s) virtual machines


LSA & Data Classes / DB Table Spaces

IN BW each InfoProvider and parts of it (e.g. fact tables/ dim tables) are assigned to data classes. Data classes refer N:1 to DB table spaces.


LSA & Data Classes / DB Table Spaces

InfoProvider data class/ table space assignment is crucial for performance and DB maintenance InfoProvider must have the correct data class assignment before first activation in productionWe can change the data class of an InfoProvider as long as it is empty In development the same data classes must exist like in production. The table spaces assignment will differ

Rules of thumb for data classes & DB table spaces in LSADomains should reside in different DB table spaces i.e. in different

data classesEDW Layers & Application Layers should reside in different DB table

spaces i.e. in different data classesPSA/ Acquisition Layer has an own table space


SAP BW Layered, Scalable Architecture-Use Adequate Storage and Access

In-memorycolumnar

Data ManagementDisc-centric

RDBMS

Reporting /

Architected Data MartsLayers

Data Warehouse Layers

Memory-centric (Ram-based) data management:By most measures, computing power doubles every couple of years.

(Moore’s Law) Exception is disk access speed it has grown only 12.5-fold in a half a century Conventional DBMS are designed to get data on and off disk quickly Memory-centric products assume all the data is in RAM in the first place RAM access speeds are up to 1,000,000 times faster than random reads on disk.

SAP NetWeaver BW


SAP BW EDW & Transparent Data ManagementUse Adequate Storage

BW &

BWA/In

Memory,columnar

DataManagement

BW &

Near Line

Storage(NLS)


SAP BW Layered Scalable Architecture- Use Adequate Storage and Access (cont.)

In-MemoryColumnar

Data Management

Disc-centricRDBMS

Leverage Memory-centric (Ram-based) data management in SAP BW Ram-based SAP BW Accelerator offers:

Performance speedup factor between 10 and 100 Compression by factor 10 Easy migration, fully transparent

Near Line Storage Interface (e.g. SAND Dynamic Near-Line Access) Full integration to SAP BW (Query access & Data Transfer Process)

V

ertic

al D

eco

mp

os

ition

C

om

pres

sio

n ~

90

%

SAP NetWeaver BW

Transparent

BW Accelerator

‘Near-LineStorage’

Reporting /Architected Data Marts

Layers

Data Warehouse Layers


RDBMSSmart data warehousing

SAP BW EDW & LSA –Consistent Architecture, Consistent Data, Consistent BI

SAP BW Transparent

Data WarehouseManagement

BW AcceleratorSmart data aggregation

& retrieval

Near-Line StorageSmart data volume

management

BW LSA

Analytic Engine

Data ManagementCross Media

Manager

Staging

Extraction


Agenda






6. Summary


Preserve a Holistic View on BI & DWHing BW EDW based on LSA is Just The Beginning….

2. It may be incremental in terms of landscape consolidation

An EDW implementation is always an incremental one, never a big-bang.

1. Incremental in terms of functional and organizational coverage:

De

ma

nd

Pla

nn

ing

Ge

ne

ratin

g D

em

an

d

Pro

cure

2 P

ay

CO

PA

..... .....

EDW

nBI Application Coverage & EDW

Org

Un

it A

Org

Un

it B

Org

Un

it C

Org

Un

it N

..... .....

EDW

nOrganizational Coverage & EDW

SourceSource

Source

Local

SAP BI Local

SAP BI

Source

Source

SourceLocal

SAP BI Unit 2SAP BI

Stream C SAP BI Group

SAP BI

Unit 1

DWH

EMEA:

Global BI

AMSAPA

…

12

3

EMEA :

Global BI

AMSAPA…


Preserve a Holistic View on BI & DWHing I. BI & DWH Consolidation Means Centralization

As a result of BI & DWH consolidation we will achieve a new corporate information culture: Awareness about the value of standards for consistency, flexibility & lower TCO

Organization: BI CCProcesses

Tools: SAP BW, BIA

Architecture & Landscape:

BW LSA/ BW EDW

Development: central BI applications template


Preserve a Holistic View on BI & DWHing I. BI & DWH Centralization - Value

BW EDW Customer LSA

Central BI Applications Template Covering standard BI needs

Central template comprises: Complete dwh management Complete front-end design Complete operational setting

Centralize BI/ DWH:Corporate-wide standardized reporting & analytics as core BI offering based on Customer LSA.

Value:consistent data & applicationsscalable applications on EDWflexibility caused by granular

EDWdriving organizational & process

alignmentreduced TCO & TCD

BI / DWH Consolidation is achieved establishing central governance for

Architecture & standards (LSA) and

Development (BI application template for standard BI/ reporting)


Preserve a Holistic View on BI & DWHing I. BI & DWH Centralization - Limits

BW EDW Customer LSA

Central BI Applications Template Covering standard BI needs

Central template comprises: Complete dwh management Complete front-end design Complete operational setting

What is the coverage of the template-based core BI offering with respect to end-user needs?

Depending on central governance, functional-area and customer industry we could expect 60-100%

There is a ‘BI gap’ that cannot be closed by centralization!

How to close this gap?

Centralization with Federation gives the holistic BI picture!What kind of Federation is reasonable depends on the gap-size.


Preserve a Holistic View on BI & DWHing II. Data Federation – Small Gap

E.g. CP-Customer Sales-force & local market data

Solution:Standard template + Data Federation/ Query Federation

Query/ Semantical Layer

Local

To cover small gaps between centralized BI offering and local BI needs think about Data Federation (Data Federator)

BW EDW

Central BI


Preserve a Holistic View on BI & DWHing II. Federation – Medium Gap

E.g. Pharma industry: considerable amount of individual country reporting

Solution:SAP Business Explorer (Newton) Appliance (based on BWA)


It does not always make sense to transfer data to the BW EDW – SAP Business Explorer (Newton) Appliances at local sides for local needs & data

BW EDW

Central BI

Country1Source

Country2Source

Country3Source

Country1 SAP Business

ExplorerServer


ExplorerServer


ExplorerServer


Preserve a Holistic View on BI & DWHing III. BW Hub & Spoke Federation

E.g. mining industry divisional reporting (Aluminum, petrol..) Prerequisite: skilled divisional IT staff Solution:BW Hub & BW Spoke architecture


It does not always make sense to transfer data to the Group BW EDW – independent divisions with own divisional BW EDW

GroupBW EDW

Central BI

Division A BW EDW

Division B BW EDW

Division C BW EDW


Agenda






6. Summary


BI-Project-Design

Life Cycle of The Customer LSA


Customer LSA : Standards - Handbook

BI-Project-DesignBI Project Design

Step 3:

Perfect Customer LSA

Step 4:

Update Customer LSA

Step 1:

Design Customer LSA

Step 2:

ApplyCustomer LSA


Customer LSA ‘Handbook’Shell


Customer LSA ‘Handbook’Land Hessen


Defining a BW / BI Excellence Framework*Reminder

BI Business Value Drivers

SAP ERP SAP CRM Other Legacy

InfrastructureEnterprise Layer Concept, Data Marts, ODS, ETL,

BI-Topology, Data Quality, Data Model

Applications and FunctionalityBI Flavors ; Reporting;

Strategic, Operational and Analytical Applications

Organization Process

StrategyBusiness Objectives, Transparency

Performance Management, Methodology

Skills, BI CompetencyCentre

BI Framework*

• Consistency• Flexibility

• Efficiency• Suitability

• Realization

• Alignment

* BI Framework introduced by Gartner


Layered Scalable Architecture (LSA) asBest Practice Modeling of High-End BWs

The LSA is a Best Practice modeling for large SAP BW Data Warehouses.

The LSA structures and standardizes a BW DWH in a transparent, service-level oriented, scalable manner.

Transparency is achieved introducing a grid on a BW DWH defined by Data Layers and Data Domains Services are modeled by Data Layers Scalability and Manageability is guaranteed by

Data Domains The broader the organizational and value-chain

coverage of a BW DWH is, the more necessary is a design-standardization like propagated by the LSA. The most comprehensive approach of a BW data logistic is an Enterprise Data Warehouse (EDW)

LSA Architectured

D a

t a

f l o

w

A transparent data-logistic like a Layered, Scalable Architecture is a ‘key success factor’

of BW ‘High-End’-/ EDW-implementations!Basic concepts can very well applied to smaller implementations!

∑


Thank you!


Further Info

[email protected]

[email protected]

Blogs:

SAP NetWeaver BW: BW Layered Scalable Architecture (LSA) / Blog Series

https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/14313

Workshop – SAP Education Germany:

PDEBW1 - BW-Blueprinting an Enterprise Data Warehouse: Architecture and Implementation Best Practices

mailto:[email protected]

mailto:[email protected]

https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/14313

http://www.sap.com/germany/services/education/schulungskatalog/project_team_training/sap-netweaver/tabbedcourse.epx?context=%5B%5B%7CPDEBW1%7C%7C1%7C%7CDE%7C%5D%5D%7C


Q&A

ContactFeedback

Please complete your session evaluation.

Be courteous — deposit your trash, and do not take the handouts for the following session.

THANK YOU !

layered scalable architecture - bi302

Documents

data warehouse layer

summary sap ag

bi data marts shouid

sap ntelligence platform

data structure dimension

soiid data foundation

b302 page 3agenda1

scalable architecture