big data has the answers: now all we need are the...
TRANSCRIPT
Big Data Has the Answers: Now All We Need Are the Questions! Steve Sonka
Page 2
Is Big Data an Economic Big Dud?
'Big data' will change the way you farm !!!
In growing field of big data, jobs go unfilled
Weather Channel Now Also Forecasts What You'll Buy
“The big-data revolution in US health care: Accelerating value and innovation”
“How big data can revolutionize pharmaceutical R&D”
Recent Headlines
Beyond Precision Ag
Monsanto Buys Climate Corp For $930 Million (10/02/13)
Monsanto Pitches Standards for Farm Data (1/31/14)
Big Data Comes to the Farm, Sowing Mistrust (2/26/14)
Page 3
Web-Enabled Toothbrushes Join the Internet of Things
Beyond Precision Ag
Wall Street Journal: March 2, 2014 10:35 p.m. ET
Page 4
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
Page 5
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
R&D
Page 6
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
R&D
First-generation
products, high price,
lots of customization
needed
Page 7
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
R&D
First-generation
products, high price,
lots of customization
needed
Mass media
hype begins
Page 8
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
R&D
First-generation
products, high price,
lots of customization
needed
Mass media
hype begins
Activity beyond
early adopters
Page 9
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
R&D
First-generation
products, high price,
lots of customization
needed
Mass media
hype begins
Activity beyond
early adopters
Negative press begins
Page 10
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
R&D
First-generation
products, high price,
lots of customization
needed
Mass media
hype begins
Activity beyond
early adopters
Negative press begins
Less than 5% of the
potential audience
has adopted fully
Page 11
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
R&D
First-generation
products, high price,
lots of customization
needed
Mass media
hype begins
Activity beyond
early adopters
Negative press begins
Less than 5% of the
potential audience
has adopted fully
Second-generation
products, some services
Page 12
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
R&D
First-generation
products, high price,
lots of customization
needed
Mass media
hype begins
Activity beyond
early adopters
Negative press begins
Less than 5% of the
potential audience
has adopted fully
Second-generation
products, some services
Third-generation products,
out of the box, product
suites
Page 13
Hype Cycle
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
R&D
First-generation
products, high price,
lots of customization
needed
Mass media
hype begins
Activity beyond
early adopters
Negative press begins
Less than 5% of the
potential audience
has adopted fully
Second-generation
products, some services
Third-generation products,
out of the box, product
suites
High-growth adoption
phase starts: 20% to
30% of the potential
audience has adopted
the innovation
Page 14
*** InfoAg 1995 ***
Beyond Precision Ag
Page 15
A mid-1990s view of precision farming from the CCNetAg group Beyond Precision Ag
Page 16
SURPRISE IS INEVITABLE:
BEING UNPREPARED ISN’T !
What’s the “it” about Big Data?
How will Big Data transform ag (if it does)?
Is it enough to know “what”-- does “why” matter?
Where’s the “easy(ier)” money?
Where is “Ag & Big Data” on the Hype Cycle?
Beyond Precision Ag
Five Questions
Page 17
The Term of Today
Beyond Precision Ag
BIG DATA
Page 18
Big Data (via Wikipedia)
Amazon.com – millions of back-end operations every day, as well as queries from more than half a million third-party sellers. Walmart – more than 1 million customer transactions
every hour – databases w/ more than 2.5 petabytes (2560 terabytes) of data – 167 times the information contained in all the books in the US
Library of Congress.
Facebook handles 50 billion photos from its user base. Windermere Real Estate uses anonymous GPS signals
from nearly 100 million drivers to help new home buyers determine commute times. Volume of business data worldwide doubles every 1.2
years. Beyond Precision Ag
Page 19
The Term of Today
Beyond Precision Ag
BIG DATA
Page 20
Dimensions of Big Data: 3 Vs and an A
Variety
Volume Velocity
Beyond Precision Ag
Page 21
Not Your Father’s Oldsmobile!
Beyond Precision Ag
Page 22
Not Your Father’s Oldsmobile!
Beyond Precision Ag
DATA !
Page 23
Not Your Father’s Oldsmobile!
Financial transactions
Movements of a cursor on a webpage
“Turns of a screw” in a manufacturing process
Tracking of web pages examined by a customer
Photos of plants
GPS locations Beyond Precision Ag
DATA !
Page 24
Not Your Father’s Oldsmobile!
Financial transactions
Movements of a cursor on a webpage
“Turns of a screw” in a manufacturing process
Tracking of web pages examined by a customer
Text
GPS locations
Your eye’s movements as
you read this text
Conversations on cell
phones
Fan speed, temperature,
and humidity in a factory
Images of plant growth
taken from drones or
from satellites
Questions Beyond Precision Ag
DATA !
Page 25
Dimensions of Big Data: 3 Vs and an A
Variety
Volume Velocity
Beyond Precision Ag
Analytics
Page 26
GIGO is Now GI nn GO
Beyond Precision Ag
Garbage In not necessarily
Garbage Out
Garbage In, Garbage Out
GIGO
Page 27
Y N
Identification of Free Parking Lots
Beyond Precision Ag
Goal: Find Free Parking Lots on UIUC Campus
Page 28
Y N
Y: Free Parking lot : no charge after 5pm
N: Not free parking lot
Identification of Free Parking Lots
Beyond Precision Ag
Goal: Find Free Parking Lots on UIUC Campus
Page 29
Results of Extended EM vs Baselines
Experiment setup: 106 parking lots of interests, 46 indeed free 30 participants, 901 marks collected
Free Parking Lot Identification
Beyond Precision Ag
Page 30
Results of Extended EM vs Baselines
Experiment setup: 106 parking lots of interests, 46 indeed free 30 participants, 901 marks collected
Free Parking Lot Identification
Beyond Precision Ag
Page 31
Results of Extended EM vs Baselines
Experiment setup: 106 parking lots of interests, 46 indeed free 30 participants, 901 marks collected
Free Parking Lot Identification
Beyond Precision Ag
Page 32
Results of Extended EM vs Baselines
Experiment setup: 106 parking lots of interests, 46 indeed free 30 participants, 901 marks collected
Free Parking Lot Identification
Beyond Precision Ag
Page 33
More Examples: Traffic Regulator Detection
Beyond Precision Ag
Unreliable “Sensors”!
Long Wait
Short Wait
Page 34
Traffic Regulator Detection
Beyond Precision Ag
Page 35
Traffic Regulator Detection
Beyond Precision Ag
Page 36
Math Formulation: A Maximum Likelihood Estimation Problem
Maximize log-likelihood by appropriate selection of truth values for claims:
1
1
1
Log-likelihood Function of EM Scheme:
log (1 ) log(1 ) log
( ; )
(1 ) log (1 ) log(1 ) log(1 )
where 1 when measured variable
M
j i j i i j iNi
em Mj
j i j i i j i
i
j
z S C a S C a d
l x
z S C b S C b d
z
is true and 0 otherwisej
Page 37
Expectation Maximization
Page 38
( ) ( ) ( )( , ) ( , , )t t tZ t j f a b d j
Expectation Maximization
Expectation Step (E-Step)
Page 39
( ) ( ) ( )( , ) ( , , )t t tZ t j f a b d j
Maximization Step (M-Step)
Expectation Maximization
Expectation Step (E-Step)
Page 40
( ) ( ) ( )( , ) ( , , )t t tZ t j f a b d j
Maximization Step (M-Step) Iterate
Expectation Maximization
Expectation Step (E-Step)
Page 41
Five Questions
What’s the “it” about Big Data?
How will Big Data transform ag (if it does)?
Is it enough to know “what”-- does “why” matter?
Where’s the “easy(ier)” money?
Where is “Ag & Big Data” on the Hype Cycle?
Beyond Precision Ag
Page 42
Five Questions
What’s the “it” about Big Data?
How will Big Data transform ag (if it does)?
Is it enough to know “what”-- does “why” matter?
Where’s the “easier” money?
Where is “Ag & Big Data” on the Hype Cycle?
Beyond Precision Ag
Page 43
Remember
the KNOWLEDGE ECONOMY !
Beyond Precision Ag
Page 44
Information Technology And Strategic Change
Jeffrey Sampler: How information technology redefines industries.
... Look at individual transactions
(1) Separabilty
(2) Aggregation potential
Beyond Precision Ag
Page 45
Transaction Characteristics (1)
Separability: extent to which information can be separated from the associated transactions
Beyond Precision Ag
Page 46
Transaction Characteristics (1)
Separability: extent to which information can be separated from the associated transactions
Beyond Precision Ag
1950s
0 or 1
Page 47
Transaction Characteristics (1)
Separability: extent to which information can be separated from the associated transactions
Beyond Precision Ag
1950s
0 or 1
Today
011010100
110111000
Page 48
Transaction Characteristics (1)
Separability: extent to which information can be separated from the associated transactions
Beyond Precision Ag
1950s
0 or 1
Today
011010100
110111000
Page 49
Transaction Characteristics (2)
Aggregation potential: extent to which the information’s value is “leverageable” beyond the original transaction
Beyond Precision Ag
011010100
110111000
011010100
110111000
011010100
110111000
011010100
110111000
011010100
110111000 011010100
110111000 011010100
110111000
Page 50
Transaction Characteristics (2)
Aggregation potential: extent to which the information’s value is “leverageable” beyond the original transaction
Beyond Precision Ag
011010100
110111000
011010100
110111000
011010100
110111000
011010100
110111000
011010100
110111000 011010100
110111000 011010100
110111000
Analytics
Page 51
How will Big Data transform ag?
Separability: extent to which information can be separated from the associated transactions
Beyond Precision Ag
Page 52
How will Big Data transform ag?
Separability: extent to which information can be separated from the associated transactions
Beyond Precision Ag
Page 53
Five Questions
What’s the “it” about Big Data?
How will Big Data transform ag (if it does)?
Is it enough to know “what”-- does “why” matter?
Where’s the “easier” money?
Where is “Ag & Big Data” on the Hype Cycle?
Beyond Precision Ag
Page 54
Is It Enough to Know WHAT?
Society will need to shed some of its obsession for causality in exchange for simple correlations:
not knowing why but only what.
This overturns centuries of established practices and challenges our most basic understanding of how to make decisions and comprehend reality
Beyond Precision Ag
Page 55
The Truth about Nutrition
The Japanese eat very little fat – suffer fewer heart attacks than do the British or Americans
The French eat a lot of fat – suffer fewer heart attacks than do the British or Americans
The Italians drink excessive amounts of red wine – suffer fewer heart attacks than do the British or Americans
The Japanese drink very little red wine – suffer fewer heart attacks than do the British or Americans
The Germans drink a lot of beer and eat lots of sausages – suffer fewer heart attacks than do the British or Americans
Beyond Precision Ag
Page 56
The Truth about Nutrition
Beyond Precision Ag
SO…………..
EAT what you want, Apparently it’s
SPEAKING ENGLISH that kills you!
Page 57
Small Data: The Hidden Key to Today’s Agriculture!
Beyond Precision Ag
Small Data – Why Big Data – What:
The “Big” Synergy!
Page 58
Five Questions
What’s the “it” about Big Data?
How will Big Data transform ag (if it does)?
Is it enough to know “what”-- does “why” matter?
Where’s the “easy(ier)” money?
Where is “Ag & Big Data” on the Hype Cycle?
Beyond Precision Ag
Page 59
Telematics
Beyond Precision Ag
Telematics connects the farm firm
Page 60
Optimal Fertilizer Distribution
Beyond Precision Ag
Page 61
Optimal Grain Movement
Beyond Precision Ag
Page 62
Where’s the Easy(ier) Money?
Beyond Precision Ag
Page 63
Five Questions
What’s the “it” about Big Data?
How will Big Data transform ag (if it does)?
Is it enough to know “what”-- does “why” matter?
Where’s the “easier” money?
Where is “Ag & Big Data” on the Hype Cycle?
Beyond Precision Ag
Page 64
A mid-1990s view of precision farming from the CCNetAg group Beyond Precision Ag
Page 65
A Big Data “Vision”
Beyond Precision Ag
Page 66
Hype Cycle(s)
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
1995 -- Precision Ag 2014 -- Big Data
Page 67
Hype Cycle(s)
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
1995 -- Precision Ag 2014 -- Big Data
2014
Page 68
Hype Cycle(s)
Technology
Trigger
Peak of Inflated
Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Expectations
Time
1995 -- Precision Ag 2014 -- Big Data
2014
20??