where is the big data industry going? from structure:data 2013
DESCRIPTION
Presentation from Sean Gourley, Quid #dataconf More at http://event.gigaom.com/structuredata/TRANSCRIPT
DATA
“What Data Can’t Do”
David Brooks Op-Ed, February 13th 2013
“Big Data has trouble with Big Problems”
- David Brooks
Advertising
Advertising Prediction
DATA:SCIENCE
Jeff Hammerbacher! DJ Patil!
Data Science
Kosinski et al PNAS 2013
Liking Curly Fries on Facebook is top predictor of intelligence
DATA:SCIENCE
DATA:intelligence
Iraq:2007
War IED
Open Source Data Collection
Iraq Fr
eque
ncy
Attack Size
xmin
α = 2.31
Iraq Fr
eque
ncy
Attack Size
xmin
α = 2.31 PREDICT?
Insurgent model: Nature 2009
Zhao et al, Phys. Rev. Lett. 2009
Impact of increasing troop numbers
DATA:SCIENCE
DATA:intelligence vs.
Data:SCIENCE Data:Intelligence
Improvement 10% 10x
Data:SCIENCE Data:Intelligence
Improvement 10% 10x
Model goal Predict/Optimize Create/Change
Data:SCIENCE Data:Intelligence
Improvement 10% 10x
Model goal Predict/Optimize Create/Change
Decision Algorithm Human
Data:SCIENCE Data:Intelligence
Improvement 10% 10x
Model goal Predict/Optimize Create/Change
Decision Algorithm Human
Data Big/Clean Small/Messy
Data:SCIENCE Data:Intelligence
Improvement 10% 10x
Model goal Predict/Optimize Create/Change
Decision Algorithm Human
Data Big/Clean Small/Messy
Communication Equations Stories
Data:SCIENCE Data:Intelligence
Improvement 10% 10x
Model goal Predict/Optimize Create/Change
Decision Algorithm Human
Data Big/Clean Small/Messy
Communication Equations Stories
Problem Tactical Strategic
Powerpoint Google Excel + +
STRATEGIC TOOLKIT
“What is the most effective way to allocate capital to spur growth in
K-12 education technology”!
**…and how will I know if we’re successful!
“What are the dominant narratives about
climate change in India”!
**…and how does it vary between age groups!
“What are my competitors doing with advanced
flexible display technology”!
**…should I compete or partner with them?!
“What is the structure of the
insurgent groups in Syria”!
**…what is the likely impact of peace keepers!
Five Heuristics
For Using Data to Solve Big Problems
1. Data needs to be designed for human interaction
Human Machine
Human Centered UI
2. Understand limits of human processing.
900 ms
Source: Nanex
3. Data is messy, incomplete and biased.
…Deal with it…
4. Data needs theory
The future will look like the past…
Build model:Understand
5. Data needs stories… stories need data
Equa%on governing Insurgent Dynamics
Mathematical
Myth:Hydra
BIG DATA & BIG PROBLEMS
DATA:intelligence
Human + Machine
Interface
Human + Machine