data vault modeling et retour d'expérience
TRANSCRIPT
![Page 1: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/1.jpg)
BÂLE BERNE BRUGG DUSSELDORF FRANCFORT S.M. FRIBOURG E.BR. GENÈVE
HAMBOURG COPENHAGUE LAUSANNE MUNICH STUTTGART VIENNE ZURICH
Agile Business Intelligence @ Evam
![Page 2: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/2.jpg)
Plan
• Introduction ( F. Kang à Birang)
• Pre-project (F. Kang à Birang & J-M. Delacrétaz)
• Agile project management (A. Martino)
• Agile architecture (E. Fidel)
• Data quality (A. Martino)
• EVAM Feedback (B. Albietz)
![Page 3: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/3.jpg)
Introduction
Fabienne Kang à Birang – Business Analyst / Product owner
![Page 4: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/4.jpg)
Introduction
• EVAM Presentation
• Project Sponsor
• Director
• Indicators
• 2013 – Existing B.I.
![Page 5: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/5.jpg)
Pre-Project Phase
Fabienne Kang à Birang – Business Analyst / Product owner
Jean-Marc Delacrétaz – Developer
![Page 6: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/6.jpg)
Pre-Project Phase
• Target
• Operational reporting
• Problems encountered @ EVAM
• Data interpretation
• Business rules errors
• Prerequisites
• Dictionary
• Population hierarchized
![Page 7: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/7.jpg)
Preexisting B.I.
• 2013
• P.O.C. to introduce B.I. «philosophy»
• Chosen Tools
• ETL : Talend
• Reporting : Tibco JasperReport
• Weaknesses
• Lack of expertise & methodology
• Bad performances
![Page 8: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/8.jpg)
Decision in August 2014
• Start from scratch
• With Trivadis Lausanne as a partner
• Tools
• Performances
• Architecture with « Best practices »
![Page 9: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/9.jpg)
Agile Project
Management
Adriano Martino – Senior B.I. Consultant
![Page 10: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/10.jpg)
Agility
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan
![Page 11: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/11.jpg)
Organisation
• Evam• Evam
• Trivadis
• Evam• Trivadis
Scrum MasterProduct
Owner
CustomerDeveloppers
![Page 12: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/12.jpg)
Agile Objectives
• Deliver working software frequently
• Adapt to change
![Page 13: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/13.jpg)
Scrum components overview
Sprint
Planning
Sprint
Backlog
Product
Backlog
Daily
Stand up
Sprint
2 to 4
weeks
Sprint
Review
Retrospective
![Page 14: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/14.jpg)
Normal Process for a B.I. need
Business
Analysis
Design of the
modelImplementation
Unit TestingVolume
testing
User
Acceptance
Testing
New
Need
Rework
Rework Rework
Rework
Deployment
to Validation
Deployment Production
![Page 15: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/15.jpg)
Normal Process for a B.I. need
![Page 16: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/16.jpg)
Agile Objectives
• Adapt to change
• Deliver working software frequently
• At regular intervals, the team reflects on how to become more effective
![Page 17: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/17.jpg)
Cadence
SCRUM
EVENT
DRIVEN
Sprint1 Sprint2 Sprint3 …
RetrospectiveReviewReleasePlanning1 2
1
3 4
2 3 4 1 2 3 4 1 2 3 4
1 2 2 2 2 213 42
![Page 18: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/18.jpg)
Agile Objectives
• Adapt to change
• Deliver working software frequently
• At regular intervals, the team reflects on how to become more effective
• Work close to business
![Page 19: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/19.jpg)
Collaborative Workshops
Business
Need
analysis
Technical
analysis
Live dev
Prototyping
Live
testing
![Page 20: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/20.jpg)
Agile B.I. Architecture
• Evolutive
• Easy change management
• Parallelisable development
• Business oriented
• Integration
• Possibility to automate generation
We choose Data Vault
Modelling
![Page 21: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/21.jpg)
Agile
Architecture
Eddie Fidel – Senior B.I. Consultant
![Page 22: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/22.jpg)
STAGING
DYNAMIC ETL
Enterprise
Data
Warehouse
With data vault
Modeling
Agile Bi Architecture
SOURCESVirtualized
Data Marts
![Page 23: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/23.jpg)
STAGING
DYNAMIC ETL
Enterprise
Data
Warehouse
With data vault
Modeling
Data Warehouse Layer
SOURCESVirtualized
Data Marts
DYNAMIC ETL
![Page 24: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/24.jpg)
What is Data Vault ?
• Data Modelling Method for Data Warehouses in Agile Environments
• Developed by Dan Linsted
• Suitable for
• DWH Core Layer
• Optimized for
• Agility / Integration / Historization
![Page 25: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/25.jpg)
Data Vault composition
• Decomposition of Source Data
• Split Data into Separate Parts
Hubs Business Entity
Links Relations
Satellites Contexts
Business Oriented
![Page 26: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/26.jpg)
Data Vault composition
• Elements : Hub – Link – Sat
Customer
Sat
Sat
Sat
CustomerProduct
Sat
Sat
Sat
Product
Hub = List of Unique Business Keys
Link = List of Relationships, Associations
Satellites = Descriptive DataOrder
Sat
Sat
Sat
Order
Link
![Page 27: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/27.jpg)
Avantages and challenges
• Standard ETL Rules to Load Data Vault
• Easy Extensibility of Data Vault Model
• Integration of Multiple Source Systems
• Traceability and Complete History
• High Number of Tables in Data Vault
![Page 28: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/28.jpg)
What does the Data Vault generator do ?
• Tables
• Indexes
• Surrogate keys
• Foreign keys
• Partitions
• Loading process
• SCD1 / SCD2
• Loading audits
• Handling Errors
![Page 29: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/29.jpg)
Generator value
29
Business spec
Technical spec
Development
Test
Deployment
Qu
ality
ass
ura
nce
Do
cum
en
tati
on
Simplify
Generator
Do
cum
en
tati
on
QS
Total Savings
Fast and short implementation cycles
Broad flexibility of change
Auto-generated quality assured components
Huge time and cost savings
On-going and recurrent with each
step of modification or enlargement!!!
![Page 30: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/30.jpg)
STAGING
DYNAMIC ETL
Enterprise
Data
Warehouse
With data vault
Modeling
Dynamic ETL
SOURCESVirtualized
Data Marts
![Page 31: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/31.jpg)
Dynamic ETL for DWH
• Parallel Loading
• HUB
• LINK et SAT
• Dynamic call to loading procedures
• No deployment of ETL needed
![Page 32: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/32.jpg)
STAGING
DYNAMIC ETL
Enterprise
Data
Warehouse
With data vault
Modeling
Dynamic ETL
SOURCESVirtualized
Data Marts
DYNAMIC ETL
![Page 33: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/33.jpg)
Data Mart
• Business Need Oriented
• Virtualized DM (materialized view)
• Can be regenerated from scratch
• Find value at a point in time
• Good perfomance
• Automatically regenerated (no deployment)
![Page 34: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/34.jpg)
Data Quality
Adriano Martino – B.I. Consultant
![Page 35: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/35.jpg)
Quality report
• Automated
• Daily execution
• Simple development
• Possible to send mail based on result
• Direction support to involve Business
![Page 36: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/36.jpg)
EVAM Feedback
Bruno Albietz – I.T. Manager
![Page 37: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/37.jpg)
Keys Learnings
• Show business value as early as possible and keep the ball rolling
• Project: December 2014 – June 2016
• Phased implementation: 1st output in June 2015, then regular outputs on a monthly basis
• Be prepared to spend most of your time on data quality
• The lifeblood of B.I. projects
![Page 38: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/38.jpg)
Keys Learnings
• Prepare knowledge transfer to your staff during the project
• Modelling, ETL, Reporting
• Good project management practice, from business requirements to report development
• Increase user buy-in with Scrum
• Key users and management involved from day 1
![Page 39: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/39.jpg)
Keys Learnings
• Learn to say “ No ”
• B.I. quality versus business process quality
• B.I. is also here to show process deficiencies, do not try to solve all business issues within the B.I. project
![Page 40: Data vault modeling et retour d'expérience](https://reader035.vdocuments.mx/reader035/viewer/2022062412/5871d21f1a28ab423c8b5db1/html5/thumbnails/40.jpg)
Q & A