from near to maturity - presentation to european data forum
DESCRIPTION
Big Data - it's the big buzz. But is it dead on arrival? In this presentation Daragh O Brien looks at the history of information management, the challenges of data quality and governance, and the implications for big data...TRANSCRIPT
![Page 1: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/1.jpg)
EUROPEAN DATA FORUMFrom Near to Maturity – Making Big Data relevant to Business
© 2013 Castlebridge Associates
![Page 2: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/2.jpg)
TrainingConsultingCoaching/
MentoringProject
Management
Quality Assured
InformationQuality
DataProtection
DataGovernance
InformationQuality
DataProtection
DataGovernance
Certified Trainers
External QA
Audits
Irish State Approved Training Provider
Quality Assured Syllabu
s
Qualified & Experienced
IQCP & DP
Certified
Certified PMs
Many Industries
Govt Edu UtilitiesFin. Svcs
Non-Profit
Telco
About Castlebridge Associates (www.castlebridge.ie)
![Page 3: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/3.jpg)
What are your expectations?
![Page 4: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/4.jpg)
HISTORYOr: How we came to have all this data anyway…
![Page 5: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/5.jpg)
Ancient Sumeria
• Written in Accadian• Used pictographic representations of information and concepts baked/carved
into tablets made of clay (high sand content)
![Page 6: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/6.jpg)
Portable Information Management
![Page 7: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/7.jpg)
Filing: The Birth of Big Data
Image by Nic McPhee @ commons.wikimedia.com
![Page 8: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/8.jpg)
6 thousand years
Tablets Tablets
Physical Data (5925 years approx.)
Electronic Data(c.75 years)
• More Information processed• Information processed faster• More ‘self service’ data
processing • Changed expectations of data and processing.
![Page 9: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/9.jpg)
But the BIG QUESTION is:
SO WHAT??
![Page 10: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/10.jpg)
Particularly as we may be too late!• Barry Devlin,
• “Big Data is Dead. It’s all just Data!!” • (B-EyeNetwork, December 2012)
• Samuel Arbesman (Wired.com)• “Stop Hyping Big Data and Start Paying
Attention to ‘Long Data’”• (Wired.com – January 2013)
• Ted Friedman (Gartner) on Twitter:
Image © Barry Devlin/B-EYENetwork
![Page 11: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/11.jpg)
Is Big Data just a matter of perspective?
![Page 12: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/12.jpg)
MATURITY
![Page 13: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/13.jpg)
General Overview
Weak
Strong
Maturity models are (almost always) 5 step models that associate common characteristics of organisations that are doing things well or are on the way to doing them better.
![Page 14: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/14.jpg)
Initial
Repeatable
Defined
Managed
Optimising
Where is Big Data?
Uncertainty
Awakening
Enlightenment
Wisdom
Certainty
(Overlaying Crosby CMM model with DMBOK Maturity model)
![Page 15: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/15.jpg)
Initial
Repeatable
Defined
Managed
Optimising
Where is Big Data?
Uncertainty
Awakening
Enlightenment
Wisdom
Certainty
![Page 16: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/16.jpg)
Maturity: Answering So What Questions
So What…
…is it?
…problems will it solve?
…will we be able to differently?
… legal / regulatory risks does all this pose?
… do we need to do to tap this gold mine?
… are we not doing today that this will enable?
… are we not doing today that this make worse?
![Page 17: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/17.jpg)
THE CHALLENGES
![Page 18: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/18.jpg)
Organisations don’t manage data well
Information Governance / Data Governance only now emerging as formal disciplines
Information Quality / Data Quality also only beginning to be coherently tackled in many organisations
Phone companies still get bills wrong
Data Protection breaches still occur• Note – this is more than just SECURITY
breaches
Data Migrations, CRM, ERP still fail
Metadata largely under-managed
![Page 19: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/19.jpg)
Estimated % of TURNOVER wasted by companies due to poor information quality
Time lost to organisations from staff rechecking information
Bottom Line Impact% of Risk Managers who see Information as “Significant” in their Risk Management plans
88%
% of Chief Financial Officers who see Information Management as a barrier to achieving Business goals 75%
30%
35%
Deloitte
Forrester
Gartner
IBM
% Data Migrations that FAIL (don’t deliver, over run time/budget, deliver reduced functionality)
84%Bloor
This is when dealing with “traditional” structured/semi-structured data..
![Page 20: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/20.jpg)
“So far, for 50 years, the information revolution has centered on data—their collection, storage, transmission, analysis, and presentation. It has centered on the "T" in IT.
The next information revolution asks, what is the MEANING of information, and what is its PURPOSE?”
Peter Drucker, Forbes ASAP, August 1998
![Page 21: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/21.jpg)
After the Hype Comes the Hangover
![Page 22: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/22.jpg)
Data Is the New Oil
Picture from NASA
Oil Slick
Water
Pic: US Coast Guard
![Page 23: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/23.jpg)
A REAL EXAMPLE
Names have been changed to protect the innocent
(and the guilty)
![Page 24: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/24.jpg)
The Pending Order Crisis of 2006
If order not completed, cannot be billed
![Page 25: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/25.jpg)
The Pending Order Crisis of 2006
OMG There’s MILLIONS of unbilled revenue out
there.This is a CRISIS!!!
![Page 26: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/26.jpg)
The Pending Order Crisis of 2006
The Sky is FALLING
![Page 27: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/27.jpg)
The Pending Orders Solution 2006
Elite Specialist Information Quality Agent
Licensed to “Fix the Data by all means necessary”
(firearms not actually used…)
![Page 28: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/28.jpg)
The Pending Orders Solution 2006
Orders for infrastructure had engineering statuses
Orders for could have multiple dependent
products – double counted
Revenue Assurance did not look at all relevant data
sources
Dependencies between process steps not
understood
![Page 29: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/29.jpg)
The Pending Order Solution 2006
There wasn’t a Crisis situation • External Factors affected order completion times
• Intra-order product dependencies lead to double counting
• Context of the process was important
Revenue Assurance Hypothesis was flawed
![Page 30: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/30.jpg)
ASKING THE RIGHT QUESTIONS
![Page 31: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/31.jpg)
One way of thinking about data
![Page 32: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/32.jpg)
Question 1: So What Data Do We Need?
No doubt that more data helps, but don’t for a minute think that you need all data to make an informed business decision.
Phil Simon To Big To Ignore: The Business Case for Big Data
Organizations that are effectively leveraging the power of Big Data realize that they will never capture all relevant information.
![Page 33: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/33.jpg)
Question 1: So What Data Do We Need?
Chicken Little © 2005 Disney Corporation
![Page 34: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/34.jpg)
Question 1: So What Data Do We Need?
What is the problem we are trying to solve?
What is the “Information Environment” for this problem?
What is the Process Context for this problem?
![Page 35: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/35.jpg)
The Pending Orders Crisis
What is the problem we are trying to solve?
• Customers are not being billed for services they have• Revenue from services is not being realised• We have orders that are not being completed
What is the Process Context for this problem?
What is the “Information Environment” for this problem?
![Page 36: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/36.jpg)
Question 1: So What Data Do We Need?
To properly answer this question you need to have:
A PLAN
![Page 37: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/37.jpg)
Question 2: So What is Stopping us doing it?
Regulation: • Data Protection Rules• Industry Regulations re: Data Governance
Technology: • Legacy architecture• Technology Management (Silos)
Human Factors: • Skills (technical/problem solving/analytical• Political (Change Management)
![Page 38: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/38.jpg)
Question 2: So What is Stopping us doing it?
Data: • Quality of internal data• Completeness, consistency, “transactability”
• Ability to link external data to internal data• Governance of data
• Decision rights• Supplier relationship management• Roles & Responsibilities
![Page 39: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/39.jpg)
Example of Regulation
Use of Location Data in Telecommunications is affected by EU Data Protection rules
Consent is required for it to be used for “Value Adding” services
Location Data
![Page 40: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/40.jpg)
Data Quality
I am incredibly sceptical about claims that “Big Data” is immune to Data Quality problems.
Statistically, Data Quality errors will skew your mean, and create outliers that affect your analysis.
While “Big Data” might not be as prone to ‘fat finger’ errors, you still have to consider whether the mechanisms gathering the data are correctly calibrated and the algorithms for analysis are running correctly or whether you have measurement errors you don’t know about.
Dr Thomas C Redman, thought leader in Data Quality
![Page 41: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/41.jpg)
Data Quality & Lineage are Key
![Page 42: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/42.jpg)
Databases are like lakes
System B
SystemA
System C
![Page 43: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/43.jpg)
Bias within the Data?
The greatest number of tweets about Sandy came from Manhattan. This makes sense given the city's high level of smartphone ownership and Twitter use, but it creates the illusion that Manhattan was the hub of the disaster. Very few messages originated from more severely affected locations, such as Breezy Point, Coney Island and Rockaway. As extended power blackouts drained batteries and limited cellular access, even fewer tweets came from the worst hit areas.
Kate Crawford Hidden Biases in Big Data, HBR 1st April 2013
![Page 44: From Near to Maturity - Presentation to European Data Forum](https://reader036.vdocuments.mx/reader036/viewer/2022070302/546e7743af795980298b56e7/html5/thumbnails/44.jpg)
Human Factors
• Bias• Politics• Skills• “Attachment Disorder”• Change & Transition Management