agile and bi a perfect matchbi-podium.nl/mediafiles/upload/agile_bi/marco... · agile . the devils...
TRANSCRIPT
![Page 1: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/1.jpg)
Agile and BI – a perfect match Agile BI congres | Marco Coopmans | 13 december 2012
?
![Page 2: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/2.jpg)
What is Agile?
![Page 3: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/3.jpg)
Why start working differently?
A software development project …
• Takes too long
• Is too expensive
• Does not deliver what is asked
• Does not deliver what is needed
![Page 4: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/4.jpg)
The Agile Manifesto
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
[Agile Manifesto, Utah, 2001]
Values
![Page 5: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/5.jpg)
The main goal of agile is:
Delivering as much value as possible for the given time and money
How?
• Only make realy needed functionality, the highest value first
• Eliminate waste (lean) – only added value
• Mitigate risks – show working, production feasible software
The goal of agile
![Page 6: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/6.jpg)
The characteristics of agile vs. waterfall
The user knows what he needs
The developer knows how to build it
During the project nothing changes
Waterfall
The user discovers during the project what he needs
The developer discovers during the project how to build it
There is a lot of dynamics and changes during the project
Agile
![Page 7: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/7.jpg)
The devils triangle - Waterfall vs. Agile
Waterfall Agile
Time
Money
Quality
Functionality
Functionality
Time
Money
Quality
Fixe
d
Var
iab
le
Fixe
d
Var
iab
le
Determ
ines
Determ
ines
![Page 8: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/8.jpg)
Agile and architecture
See the curve coming, but don’t overdrive
your headlights
![Page 9: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/9.jpg)
• Keep it simple – As simple as possible
• Eliminate waste – Don’t carry any waste as long as it is
not necessary
– Don’t invest (too much) in the future
• Postpone choices as long as possible – Do things only when they are really
needed
Some architectural agile principles
Agile is delivering working software as quickly as
possible (business value)
Be a minimalist
![Page 10: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/10.jpg)
• Architecture is not determined at the beginning,
but “emerges” (form follows function)
• Determine a start architecture: a (minimal) set
of guide lines
• That gives a lean-and-mean
framework that
grows during development
Emerging architecture
Refactoring is a fact!
![Page 11: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/11.jpg)
Agile is not short term solutions
To avoid a dead end and minimalize the impact of
refactoring you have to look ahead,…
but not too far!
An experienced
SCRUM team will not
let this happen!
![Page 12: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/12.jpg)
SCRUM is the most applied form of agile
![Page 13: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/13.jpg)
The SCRUM process
Grooming/Poker sessions
Sprint Backlog
Demo
Sprint (2 to 4 weeks)
Production feasible Product/software
Feedback, new input
Product Backlog
Priority Sprint planning
•Product Vision •Solution outline
(architecture, tooling) •Business Case / ROI
•Release planning (1st release = MVP)
Development
Daily
stand up
Re
leas
e 1
R
ele
ase
2
Re
leas
e x
R
ele
ase
..
![Page 14: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/14.jpg)
SCRUM Roles and Organisation
Product Owner
Scrum Master
Stake Holders Scrum Team
Tune Tune
Supports
Project Team
![Page 15: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/15.jpg)
Roles and Organization – Product Owner
Is a delegate of the business who has the mandate to determine what is on the product backlog and the prioritization of it
Holder of the product vision
Owner of the product backlog and user stories (requirements)
Represents all stake holders (business, IT, external, …)
Accepts or rejects the product of a sprint
Decides release date and content
Product Owner
Product Owner
Product Owner sits
in the driver seat
![Page 16: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/16.jpg)
Roles and Organization – Scrum Team
A multidisciplinary team (designers, developers, testers, architects, …) – all expertise necessary to deliver a production feasible product
Design, develop and test the user stories
Is self-organizing, without hierarchy
Determines, together with the Product Owner, the content off a sprint and commits itself to it!
Scrum Team
Scrum Team
Scrum Team is self-
organizing
![Page 17: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/17.jpg)
Roles and Organization – Scrum Master
Facilitates the SCRUM process of the team
Ensures optimal conditions for the productivity of the team
Solves impediments (blocking issues)
Is part of the team, … but is supporting (has no authority)
Scrum Master
Scrum Master
Scrum Master is a
facilitator
![Page 18: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/18.jpg)
Roles and Organization – Stake Holders
Everybody who has an interest in the product (endusers, managers, security officers, architects, system administrators, controllers, supervisors, …)
Are being represented by the Product Owner
Stake Holders don’t have direct access to the Scrum Team! – takes place via the Product Owner
Stake Holders
Stake Holders
![Page 19: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/19.jpg)
Agile and BI
![Page 20: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/20.jpg)
Why start working differently?
A software development project …
• Takes too long
• Is too expensive
• Does not deliver what is asked
• Does not deliver what is needed
Does this also apply for BI projects?
According to Gartner research estimation is that
70 to 80% of BI projects fail. And the number
one reason cited for this disturbing rate of
debacle? Poor communication. (Patrick Meehan, President and Research Director at Gartner, 2011)
![Page 21: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/21.jpg)
Agile BI is about tools?
Agile BI = Delivering functionality as soon as
possible
Agile BI = Self-service
Agile BI = Tools
It’s not only
about
delivering
data fast, it’s
also about the
right data and
quality
![Page 22: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/22.jpg)
Agile BI is only suited for the front-end?
Back-end
Front-end
Waterfall
Agile
Only 20/30% would be agile!
70/80%
![Page 23: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/23.jpg)
Agile BI is layer driven?
Sprint 1
Sprint 2
Sprint 3
Sprint 4
Sprint 5 First visible result in sprint 5
![Page 24: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/24.jpg)
Fro
m s
ou
rce
to
info
rmat
ion
business value
Agile is delivering business value (each sprint)!
But how are you
gonna do this every
sprint, from the
beginning?
![Page 25: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/25.jpg)
The minimal architecture
Choose a minimal set of components to achieve the requested functionality. This depends on tools, source systems, transformation, integration, …
![Page 26: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/26.jpg)
2 ways - #1 Report driven (slicing)
A report is growing with
each sprint
This will need a few sprints to get a sustainable report to get efficient feedback
![Page 27: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/27.jpg)
2 ways - #2 Data / information driven (refining)
Reports are refining /
maturing with each
sprint
![Page 28: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/28.jpg)
The ‘Mona Lisa model’ – from raw data to information
Production Base
Dimensional model (Facts and Dimensions)
Initiate & Model Storming
Making accessible
Refine Raw
ETL
Universe(s)
Standard reports
Experienced Domainexperts
User documentation
System- & Operatingdocumentation
Trained key-users
Sprint … Sprint …
Sprint … Sprint …
Datastorage (DWH/DM/…)
Unlock as much as possible (raw)
data
Refine content: - Refine Dimensional
model - Refine Universe - Make basic
report(s) - Add business rules,
transformations, integrations, …
- Add history - ….
Refine to end users:
- Make final rapports - Add security/
authorization - Finalize metadata - Finalize
documentation - ….
Bring to Production
Set op a base version
(dimensional) information-
model
Confronting users with data as soon as
possible
![Page 29: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/29.jpg)
Organisation – BICC and SCRUM team
IV IT-supplier Business
Business Intelligence Competence Center
Legacy Designer/Developer
Technical Consultant (dba, network, infra, etc)
ETL Designer/Developer
BI/Warehouse Consultant
BI Tool Specialist
Power User
Key-User
Proces/Data owner BICC manager
BI Business Modeler (BI Tool Specialist)
Enterprise WH Architect
BI Domain Expert
Management Reporting Authority
End-User
BI/ETL Designer/Developer
Production Guard
Service Coördinator
Project Team
Product owner
Scrum master
SCRUM Team
BI/ETL Designer/Developer
Enterprise WH Architect BI Domain Expert BI Business Modeler Developer/ Specialist Power User
Business IT
![Page 30: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/30.jpg)
BI issues Agile
Why agile is so well suited for BI
Unclear information need Agile assumes that requirements are not clear and/or change, that is part of the approach. By a constant feedback requirements become clear.
Management / business involvement
Business is in the lead (Product Owner). Business is part of the team (domain experts and power users). Stake holders are being involved through demo’s, grooming sessions, sprint plannings.
Quality of data Quality of data is being tackled at the moment the issue arrises. The issue is made clear from a concrete information need and is better negotiable than.
Definitions Definitions are being established during development (by the business), when it is necessary, based upon real data/visible results.
Reliability of information Reliability is being guaranteed by the participation of domain experts and power users in the SCRUM team (not by IT). They also give the demo!
Organisation A SCRUM team fits realy good with a BICC. The business (power users, etc.) will grow into their role during the project.
![Page 31: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/31.jpg)
Agile and BI are a perfect match!
![Page 32: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/32.jpg)
Preconditions and Pitfalls
![Page 33: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/33.jpg)
Preconditions - Culture
• Give confidence
• Learning organisation - making mistakes is allowed
• Transparancy - no judgement
• Entrepenearship and
creativitity
• Belief
Agile means a change of culture
![Page 34: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/34.jpg)
Preconditions – Process en organisation
• Committed Product Owner, with
mandate
• Sufficient business (customer)
participation – Preferably Power Users
• IT Professionals (craftsmanship)
• (SCRUM) process is being
followed – meetings, stand-ups etc. are not for
discussion
The business sits in the driver seat
![Page 35: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/35.jpg)
Preconditions – Technique
• Enough complexity
• Emergent architecture
• Continuous build and
deploy
• Automated testing
• Production(like) data available
• The SCRUM team determines the solution – no
one else
![Page 36: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/36.jpg)
Pitfalls - Sprint
• No ‘closed’ sprint – New features, requirements added during
the sprint
• No real commitment for a sprint
• Laying the bar too high – Don’t do too much at the same time
• Laying the bar too low – Don’t be satisfied too easily
• Too much production-oriented – Communicating is production! It’s not
about the amount of code
– 10 to 30% preserved for grooming etc.
• No Definition of Done (DoD)
![Page 37: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/37.jpg)
Pitfalls - Solution
• Holding on to standard
solutions, architectures – New techniques provide many
opportunities, try them
• Change to generic / “flexible”
solutions too quickly – Generic datamodels, Datavault, …
• Too little focus on quality – short
term solutions
![Page 38: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/38.jpg)
Pitfalls - Leadership
• Agile is not about working late
(on the contrary)
• Fall back on classical
management – The Team is self-organizing, even at “bad
weather”
It’s not about working harder
but working smarter
![Page 39: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/39.jpg)
Pitfalls – Client/Business
• No clear business objective, lack
of product vision – Why are we doing this? What are we trying
to achieve?
• Insufficient coordination with
stakeholders (by the Product
Owner) – Lack of support
![Page 40: Agile and BI a perfect matchbi-podium.nl/mediaFiles/upload/agile_bi/Marco... · Agile . The devils triangle - Waterfall vs. Agile Waterfall Agile Time Moneyes ... Some architectural](https://reader034.vdocuments.mx/reader034/viewer/2022051812/602b8488422d3422d960b25c/html5/thumbnails/40.jpg)
Conclusion
BI and Agile are a perfect match,
but ….
It’s a difficult game to play