when every shot counts - can you make a difference or will you fail
DESCRIPTION
Intelligence for corporate survival is a how to guide for the changing Big data world. It can mean the difference between life and deathTRANSCRIPT
Slide 1
By
Paul Ormonde-JamesTwitter: @pormondejamesLinkedin http://au.linkedin.com/in/ormondejames
When every shot countsWhen every shot counts
Intelligence for Corporate Survival
Markus Evans
Consumer Intelligence & Analytics
Conference
Park Hyatt, Melbourne, Australia
22-23 August 2013
Slide 2Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
s
We live in a digital
world
Slide 3Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
sUbiquitous Computing
More data from more machines
Slide 4Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
sVirtualisation
Storage getting
cheaper by the minute
Slide 5Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
sCloud Computing
Data now physically all over
the world and growing at an increasing rate
Slide 6Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
sSocial Media…. BIG
DataMore people
communicating with more people, social data
increasing
Slide 7Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
s
InformationOverload
Slide 8Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
8
Needle in a hay stack
Finding the right customer data is likeLooking for a needle in a hay stack
Slide 9Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
sHow do we makes sense of all this???
Slide 10Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
tsThe Challenge?
How do I find the right
customers, at low cost to
make millions?
1 OFFER = 1 ACCEPT
Slide 11Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
How do companies understand WHO to market to?? CONTACT
How do companies approach YOU? CHANNELS
How do companies understand YOUR needs SEGMENTS
I’m an individual
I’m an individual
I’m an individual
I’m an individualI’m an
individual I’m not
Slide 12Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
12
The Ying & Yang
Data Quality
Analytics
TODAY I will address
Slide 13Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
13
My Experience
Data Quality
Insight Services Group
PostConnect
Data & InsightsAustralia’s largest fact based address & lifestyle profile database
• Postal address File• AMAS data• National Change of Address data• Movers data• Australian Lifestyle survey• eProducts & insights• Parcels insights >>
Slide 15Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
s
Bringing insights from data across the whole of Australia Post
Retail insights
Geospatial insights
Address insights
Parcel insights
Movers insights
Campaign insights
Slide 16Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
16
My Experience
Analytics
Slide 17Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
17
The World Bank
Head of Global Business Intelligence, The World BankWashington DCUSA
• Manage modeling & analysis teams for global impacts on banking sectors
• Managed global teams collecting data from 182 countries• Managed predictive analysis teams on portfolio risks &
cash flows• Built & managed Global teams for end to end data
control & analysis
Slide 18Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
18
The Analytic 4 P’s
PEOPLE
PLAT
FORM
S
CUSTOMER
PRO
CESSES
PERFORMANCE
Slide 19
AnerdLYTICS – The human face of numbers
and analysis
Slide 20Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
sAna Lytics
Ana
Ana Lytics – Power through people
Slide 21Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
sBusiness Intelligence – Simple form
Slide 22Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
s
Build the technology like pieces of a jigsaw
puzzle
22
Slide 23Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
23
My Experience
Data Quality
Slide 24Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
tsData Quality imperitives
24
• The data is consistent - • The databases are well designed - A well-designed database will
perform satisfactorily for its intended applications, it is extendible and it exploits the integrity capabilities of its DBMS.
• The data is not redundant - • In actual practice, no organization has ever totally eliminated
redundant data. In most data warehouse implementations, the data warehouse data is partially redundant with operational data. For certain performance reasons, and in some distributed environments, an organization may correctly choose to maintain data in more than one place and also maintain the data in more than one form.
• The redundant data to be minimized is the data that has been duplicated for none of the reasons stated above but because:
| Copyright – Paul Ormonde-james 2013
Slide 25Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
25
• The creator of the redundant data was unaware of the existence of available data.
• The redundant data was created because the availability or performance characteristics of the primary data were unacceptable to the new system. This may be a legitimate reason or it may also be that the performance problem could have been successfully addressed with a new index or a minor tuning effort and that availability could have been improved by better operating procedures.
• The owner of the primary data would not allow the new developer to view or update the data.
• The lack of control mechanisms for data update indicated the need for a new version of the data.
• The lack of security controls dictated the need for a redundant subset of the primary data.
– In these cases, redundant data is only the symptom and not the cause of the problem. Only managerial vision, direction and a robust data architecture would lead to an environment with less redundant data.
| Copyright – Paul Ormonde-james 2013
Slide 26Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
• The data follows business rules - • As an example, a loan balance may never be a negative number.
This rule comes from the business side and IT is required to establish the edits to be sure the rule is not violated.
• The data corresponds to established domains - • These domains are specified by the owners or users of the data.
The domain would be the set of allowable values or a specified range of values. In a human resource system, the domain of sex is limited to "male" and "female." "Biyearly" may be accurate but still not an allowable value.
• The data is timely - • Timeliness is subjective and can only be determined by the users
of the data. The users will specify that monthly, weekly, daily or real-time data is required. Real-time data is often a requirement of production systems with online transaction processing (OLTP). If monthly is all that is required and monthly is delivered, the data is timely.
Slide 27Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
• The data is well understood - • It does no good to have accurate and timely data if the users
don.t know what it means.
• Naming standards are a necessary (but not sufficient) condition for well-understood data.
• Data can be documented in the meta data repository, but the creation and validation of the definitions is a time-consuming and tedious process.
• This is, however, time and effort well spent.
• Without clear definitions and understanding, the organization will exhaust countless hours trying to determine the meaning of their reports or draw incorrect conclusions from the data displayed on the screens.
Slide 28Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
28
The data is integrated - • If the data is integrated, meaningful business information can
be readily generated from a combination integration generally requires the use of a common DBMS.
• There is an expectation (often unfulfilled) that all applications using the DBMS will be able to easily access any data residing on the DBMS.
• An integrated database would be accessible from a number of applications.
• Many different programs in multiple systems could access and, in a controlled manner, update the database. Database integration requires the knowledge of the characteristics of the data, what the data means, and where the data resides.
• This information would be kept in the meta data repository.
| Copyright – Paul Ormonde-james 2013
Slide 29Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
29
• The data satisfies the needs of the business -
• The data has value to the enterprise. High quality data is useless if it's not the data needed to run the business. Marketing needs data on customers and demographic data, Accounts payable needs data on vendors and product information.
• The user is satisfied with the quality of the data and the information derived from that data -
• While this is a subjective measure, it is, arguably, the most important indicator of all. If the data is of high quality, but the user is still dissatisfied, you or your boss will be out of a job.
• The data is complete - • All the line items for an invoice have been captured so
that the bill states the full amount that is owed. All the dependents are listed for an employee so that invoices from medical providers can be properly administered.
| Copyright – Paul Ormonde-james 2013
Slide 30Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
30
• There are no duplicate records - • A mailing list would carry a subscriber, potential buyer or
charity benefactor only once. You will only receive one letter that gives you the good news that "You may already be a winner!“
• Data anomalies - • From the perspective of IT, this may be the worst type of
data contamination. A data anomaly occurs when a data field defined for one purpose is used for another.
• For example, a currently unused, but defined field is used for some purpose totally unrelated to its original intent.
• A clever programmer may put a negative value in this field (which is always supposed to be positive) as a switch.
| Copyright – Paul Ormonde-james 2013
Slide 31Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
31
My Experience
Analytics
Slide 32Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
32
• “Analytics is the combustion engine of business, and it will be necessary for organizations that want to grow, innovate and optimize efficiency,”
• “Given its far-reaching impact, it is one of the few software markets that thrive even in adversity.”
• Gartner analyst Rita Sallam
| Copyright – Paul Ormonde-james 2013
Slide 33Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
tsDecline of the Business Intelligence Stack
33
• Mega vendors IBM, SAP, Oracle, Microsoft and SAS still own the dominant share of the business intelligence market. SAP leads in business intelligence platform revenue
• Mega vendors own two-thirds share today versus one-third in 2007. But the area of analytics is still an open playing field. Data discovery vendors such as QlikTeck, Tableau and Tibco Spotfire grew more than the others
• “A stack-centric mentality and single vendor standardization policies won't cut it,” she said. “Understand that your organization needs a portfolio of analytic capabilities.” Gartner
• Companies now differentiating between the idea of selecting a single business intelligence vendor and establishing business intelligence standards.
| Copyright – Paul Ormonde-james 2013
Slide 34Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
34
• Standardization is becoming less about using the same toolset and more about employing specific tools, metrics, and processes for certain capabilities and use cases.
• This shows up in a decline in the number of users identifying vendors as their business intelligence standard.
• All the mega vendors experienced a drop, some by as much as 19 percent in only three years.
| Copyright – Paul Ormonde-james 2013
Slide 35Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
tsDemographics Drive Data Discovery
35
• Another area that is changing the business intelligence market is demographics.
• Millennials (ages 20 to 30) now comprise 20 percent of the workforce, but their ranks will swell to 40 percent by 2020.
• “The graduating high school class of 2011 spent all of their school years with pervasive access to the Internet – they don’t know a world without information at their fingertips,” said Sallam.
• “You tell them to go to the library to use the card catalog, and they look at you if you told them to go use an abacus to calculate the square root of 1,058.”
• These younger employees are driving the consumerization of IT, which includes how business intelligence is delivered. They want business intelligence to be as intuitive, social and collaborative as the tools in their personal life.
| Copyright – Paul Ormonde-james 2013
Slide 36Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
tsDemographics Drive Data Discovery
36
• Thus traditional reporting and ad hoc query are flat or declining, whereas data visualization in dashboards and interactive visualization are experiencing growth. “BI is becoming more visual as data discovery needs intensify,” Sallam said.
• The data discovery market will rise from $591 million currently to $1 billion by 2013, Sallam said.
• Data discovery tools typically include a mix of in-memory analytics, data mashup capabilities, dashboards, self-service delivery, light footprint and speed of deployment. These tools have become so popular that established vendors are copying them.
• Demographics are also contributing to the growth of mobile business intelligence. Gartner predicts that 33 percent of analytics will be consumed on handhelds by 2013.
• By the end of this year, 55 percent of organizations using business intelligence either have or plan to deploy mobile BI. As a result, mobile BI projects will outnumber traditional workstation projects by four to one within three years. This has everything to do with competitive advantage.
| Copyright – Paul Ormonde-james 2013
Slide 37Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
tsThe Rise of Predictive Analytics
37
• Gartner surveys show most users still focus on measurement of the past, with only 13 percent of users making extensive use of predictive analytics. Less than 3 percent use prescriptive capabilities such as decision/mathematical modeling, simulation and optimization.
• Gartner advises organizations to develop a plan to support new data volume, variety and velocity requirements. By being able to correlate, analyze and present insights from structured and unstructured information, organizations will be able to personalize customer experiences and exploit new opportunities.
• “Those that can do advanced analytics on top of Big Data will grow 20 percent more than their peers,” said Sallam. “The explosion of data volume, as well as its variety and velocity, will enable new, high-value advanced analytic use cases that drive growth and productivity.” | Copyright – Paul
Ormonde-james 2013
Slide 38Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
38
How is it done
Slide 39Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
tsThe how is important
• Regardless of methodology, most processes for creating predictive models incorporate the following steps:– 1. Project Definition: Define the business objectives and desired
outcomes for the project and – translate them into predictive analytic objectives and tasks. – 2. Exploration: Analyze source data to determine the most appropriate
data and model – building approach, and scope the effort.– 3. Data Preparation: Select, extract, and transform data upon which to
create models. – 4. Model Building: Create, test, and validate models, and evaluate
whether they will meet – project metrics and goals. – 5. Deployment: Apply model results to business decisions or processes.
This ranges from – sharing insights with business users to embedding models into applications
to automate – decisions and business processes.– 6. Model Management: Manage models to improve performance (i.e.,
accuracy), control access, promote reuse, standardize toolsets, and minimize redundant activities.
Slide 40Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
sWhat does research show us
Slide 41Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
s
Slide 42Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
s
Slide 43Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
• Barriers to Usage. A host of barriers can prevent organizations from venturing into the domain of predictive analytics or impede their growth. This “analytics bottleneck” arises from: – 1. Complexity.
• Developing sophisticated models has traditionally been a slow, iterative, and labor intensive process.
– 2. Data. • Most corporate data is full of errors and inconsistencies but most
predictive models require clean, scrubbed, expertly formatted data to work.
– 3. Processing Expense. • Complex analytical queries and scoring processes can clog
networks and bog down database performance, especially when performed on the desktop.
Slide 44Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
• 4. Expertise. – Qualified business analysts who can create sophisticated models
are hard to find, expensive to pay, and difficult to retain.
• 5. Interoperability. – The process of creating and deploying predictive models
traditionally involves accessing or moving data and models among multiple machines, operating platforms, and applications, which requires interoperable software.
• 6. Pricing.
– The price of most predictive analytic software and the hardware to run it on is beyond the reach of most midsize organizations or departments in large organizations.
Fortunately, these barriers are beginning to fall, thanks to advances in software, computing, and database technology.
Slide 45Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
45
We are known for our unique offering – enabling
smarter connections.
Slide 46Copyright 2005 Paul Ormonde-James, [email protected]
Every
shot
coun
tsEvery
shot
coun
ts
Pa
ul O
rmo
nd
e-ja
me
s
Conclusions.....
46
Slide 47
ByPaul Ormonde-JamesTwitter @pormondejamesLinked inEmail: [email protected]
When every shot countsWhen every shot counts
What will you do to differentiate?