meta data driven edw development with information server at ericsson some reflection from the real...

Meta Data Driven EDW Development with Information Server at Ericsson

- Some Reflection from the Real World

Anders Bergdahl

Affecto, Leading EIM in the Nordics

Affecto has1100 Employees, 160 in Sweden.

133 MEUR revenue, 10,5 MEUR in profit

Four competence areas:

• DM/IM

• DW/BA

• CPM

• CBI EIM

2

EDW2.0 Summary | Ericsson Internal | , Rev | 2014-03-10 |

› BI & Analytics is a set of capabilities that enable and support business processes with operational, tactical and strategic insights.

› BI & Analytics provides current, historical, predictive and prescriptive views of business operations.

Ericsson BI & Analytics Description


EDW 2.0

1. EDW 2.0 is the Next Generation DataWarehouse for Business Intelligence and Analytics at Ericsson.

2. EDW 2.0 follows LEDW(Logical Enterprise DataWarehouse) architecture pattern. It combines the strengths of traditional repository warehouse(EBW) with data virtualization, in-memory data processing and distributed processing.

3. Main features for EDW 2.0 are agility and lower cost/demand, improved reuse, data quality measurement, better data quality delivered faster, Reduced data latency, alignment with Ericsson global process and Enterprise Information model.

4. EDW 2.0 enables all information delivery and use scenarios of Standard reporting, ad-hoc reporting, visual exploration, advanced analysis and collaboration.


EDW2.0 Features

1. Harmonized and consistent definitions for measures, KPIs and dimensions.

2. Alignment of analytics data modeling with enterprise business process models and enterprise information models.

3. Established ownership of analytics products.4. Agreed process for handling cross dependencies

between different analytics products.5. Agile development process.6. Data quality measurements and reporting.


EDW2.0 Features

7. Information lifecycle management and classification of data as hot warm or cold.

8. Seamless transition from consolidated cross process analysis to localized detailed reporting on operational data.

9. Access by business process roles – improved information delivery channels.

10.Support for different data latency needs of near real time, several times a day, daily, weekly, monthly and yearly snapshots.


EDW2.0 Features

11.Search on metadata to find the reports for a given set of measures and dimensions, documentation, usage instructions, roles required, contacts for further instructions or change requests.

12.Usage monitoring of reports and data to identify unused content and usage patterns.

13.Test environment with production like data.


EDW2.0 Design Principles

› Extract once and deploy many.– Reduce load on operational systems and keep data synchronized.

› Reuse.– Reuse data, definitions, business rules, programs.

› Data near processing.– Move algorithms to data and not data to algorithms

› Read less, move less– Design to ensure less data is read and less data is moved between

processing units and layers.


EDW2.0 Reference architecture


Access Control

Data Integration (Structure, Unstructure, Mutiple S ource, Log Processing etc.)

Com

mon Inform

ation Model (C

IM) &

Business G

lossaryITK Web

Logs

Business Process/Applications (Tools & Sources Syte ms)

ONE HRMS

ITM

PRIM

GSSO

CRM360 Verdi SRM

SMA

BarTrack

Test net CRM COPAELIS

IBMS Feature Store

SMS

MHWeb

PQATCloud

Web Clients, Mobile, Desktop Clients,

Planning & Simulation Statistical AnalysisCollaboration BI

Standard Reporting

Ad Hoc Reporting

Self Service Federation/VirtualizationBroadcastingDATA PRESENTA

TION +VISUALIZA

TION

USER PROVISION

ING + ACCESS CONTROL

ENTERPRISE

DW

INTEGRATION

DATA SOURCES (EXAMPLES

)

DATA MANAGEME

NT+

DATA ARCHITECT

URE

FINANCE SUPPLY and LOGISTICS

Common Dimension / Master Data (3600 View)

Cross Data Analytics (E2E View)

EDW2.0 Capabilities MapA

ligned Processes &

Data

ILM D

efinitions

Definitions &

D

escriptions

Cross F

unctional Mapping

OPCM

Alignment Acquisition Real Time Change Data Capture Quality Governance

Profiling Cleansing Transformation Enrichment

MARKETING and SALESHR

PRODUCT MGMT PRODUCT DEV SERVICE DELIVERY IT & COMMON FUNCTIONS


Functional Capability AREAS

Product Development

Marketing & Sales

Supply & Logistics

Resource Management

Communications

Sourcing

Product Management

HR & O

Finance

IT

Security

Service Delivery

Operational Development

Real Estate

Technology and Research


LINKs to the Analytics ROADMAPS for CROSS PROESS VALUE FLOWS

O2C – Order to Cash

T2C – Tender to Cash

I2R – Incident to Resolution

Q2C – Quote to Cash

N2S – Need to Solution

O2S – Opportunity to Solution

O2F – Order to Fulfillment


Technical Capability AREAs

Technology and Data Platform

Development Frameworks

COE

Operations Support


HANASAPBW

IBM Neteeza

Microsoft SQL Server

(PDW)Hadoop

EDW2.0 – multiple technology environment

ITK

Business Process/Applications (Sources Sytems)

ONE HRMS

ITM

PRIM

GSSO

CRM360

OPCM

Verdi MDM

SMA

BarTrack

Test net ELIS

IBMS Feature Store

SMS

MHWeb

PQAT Manual

BI Semantic & Information Delivery

Logical Enterprise Data Warehouse

Data virtualization for low volume joins

Physical data movement from sources

Physical data movement across DW platforms

CDW Part Int Sol

Sandbox CDW Part Int Sol

SandboxCDW Part Int Sol

Sandbox


LEDW

› LEDW stands for “Logical Enterprise Data Warehouse” and is an enabler for an common information layer that can service the whole enterprise

LEDW

Common Data Warehouse (CDW)

An enterprise wide repository of reusable information objects,

integrated across business processes & sources, stored

in a unified schema

Partially Integrated Solutions

Can consume data from CDW but will not share data to CDW.

Sandbox

Temporary workspace for pilots / poc


Data Management Platforms

SAPBW

• Maintainence of existing solutions• Not to be used for new solution areas

HANA

• Primary platform for EDW2.0• Data originating from SAP• Data which should be combined with SAP data• Complex R&A Support• Operational Reporting• Master data hub• Platform for high performance in memory

data discovery exploration and visualization


Data Management Platforms

Netezza

MS PDW

• Very high volume data• Platform for advanced analytic usage • Use case driven usage

• Very high volume data• Use Case driven usage

Hadoop

• High volume, velocity and variety of data processing • More then 100 TB / year of data volume needed• Use Case driven usage

SQL Server

• Low volume data• Stand alone application usage


content

› LEDW introduction› LEDW development process


› Decided on IIS late 2012

› Started to develop a framework spring 2013– Support from IBM and IBM partners– Need to adapt to internal standards and ”traditions”– Big company, many views..

› Continuous work, first iteration available for use…– Still waiting for first solution in full production

The task.. In brief


EDW development - aligned to the Need 2 solution WORK flow

DescribedSolution

Realized Benefit

AcceptedDelivery

ClarifiedNeed

Phases where EDW solutions is defined, developed, tested and deployed


Business representative

UnderstandInformation model

Described solution WORK flow

Define Described Solution deliverables

Clarified Need

Describe Solution activity plan

Understandprocesses model

Model Logical Solution

Evaluate and DescribeSolution

ProposedSolution

SpecifyRequirements

PerformBusinessFulfillment Analysis

Business Case

N2S Decision

Point

N2S Decision

PointDescribed Solution

N2S Decision Points (as part of the value flow)

SODRequirements defined

Process models

LogicalInformationModels

Data quality analysis

Affected Capabilities

Prototype

Logical Solution

Source discovery & data quality assessmment

GAP analysis

DescribedSolution

Realized Benefit

AcceptedDelivery

ClarifiedNeed

TPM

Solution Area functional expert

Data modeler

SOD

Update business glossary


Accepted deliverywork flow

Develop Solution

DescribedSolution

Plan Delivery

DeliveryAssignments

UATApproval

ApprovedCut-Over

Plan

Solution In Operation

N2S Decision

Point

Approved Solution

Develop Business Solution

Define Realize Solution

Deliverables

Realize Solution

Activity Plan

N2S Decision Points

Agreed Plan for Realize Solution

DescribedSolution

Execute on Business Fulfillment

Business Case Report

Business Preparedfor New Solution

Updated Benefit Realization

Plan

N2S Decision

PointAcceptedDelivery

Implement Solution in Business

Solution In BusinessOperation

DescribedSolution

Realized Benefit

AcceptedDelivery

ClarifiedNeed

Develop Solution

PrepareRelease

Deploy Solution

LEDW development flow

Step1

Step 2

Step 3

Step 4


1. Plan delivery› Select delivery approach

– Development Project Delivery Model (see IT Project Framework following ePROPS)

– Established Capability Delivery Model (ECDM)– CDW development carried out via Establised capability model– Non integrated solution can be delivered via Development project model

› Set up a plan for the delivery– Project delivery Model (IT Project framework)– Established capability DM

› Backlog prioritization› Sprint planning

› Plan the releases– Monthly & weekly releases

Weekly Weekly Weekly Weekly Weekly WeeklyMonthly Monthly


2. Develop solution

Backlog

Sprint 1

Backlog

Sprint 2

Backlog

Sprint 3

Sprint 1

Sprint 2

Sprint 3

Production deploymentRelease R14-01


QA

QA

QA

QA


Sprint 4


Backlog

Sprint 4

QA Executed by LEDW release team

SprintExecuted by LEDW development project or established capability


› Detailed requirements› Design & prototype solution› Build solution & unit test› System integration test› Performance test› User acceptance test› Document solution

2. Develop solutionETLData model

SecurityInformation delivery (BI)


Accepted deliveryWORK flow

Develop Solution

DescribedSolution

Plan Delivery

DeliveryAssignments

UATApproval

ApprovedCut-Over

Plan

Solution In Operation

N2S Decision

Point

Approved Solution

Develop Business Solution

Define Realize Solution

Deliverables

Realize Solution

Activity Plan

N2S Decision Points

Agreed Plan for Realize Solution

DescribedSolution

Execute on Business Fulfillment

Business Case Report

Business Preparedfor New Solution

Updated Benefit Realization

Plan

N2S Decision

PointAcceptedDelivery

Implement Solution in Business

Solution In BusinessOperation

DescribedSolution

Realized Benefit

AcceptedDelivery

ClarifiedNeed

Develop Solution

PrepareRelease

Deploy Solution

LEDW development flow

1 2 3 4


3. Prepare release

PROD

Test Domain Prod Domain

NZ HANA PDWNZ HANA PDW

Data integration Data integration

System runnning production

Copy of production(reduced fact tables)Regression testUAT

Controlled by LEDW release team only

System testUser testSIT

Accessible by project and established capability resources

DevelopementEnvironmentUnit test

DEV TEST QA

ReleaseHandover

Landing area Landing area

Data integration

DEV TEST DEV TESTDEV TEST QA PROD QA PROD QA PROD


4. Deploy solution

NZ PDW

Data integration

Releases deployed by LEDW release team only

QA Prod QA Prod

Source Source Source Source Source

Landing area

PRODQA

Release R14-01 Release R14-01

Prod Domain

Data integration

meta data driven edw development with information server at ericsson some reflection from the real...

Data & Analytics