powerpoint presentationdownload.microsoft.com/documents/hk/technet/techdays2013...batch processing...
TRANSCRIPT
![Page 1: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/1.jpg)
![Page 2: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/2.jpg)
![Page 3: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/3.jpg)
![Page 4: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/4.jpg)
1,000,000,000,000,000,000,000 bytes = 10007 bytes = 1021 bytes
1 Zettabyte = 1 billion terabytes
![Page 5: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/5.jpg)
![Page 6: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/6.jpg)
![Page 7: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/7.jpg)
Internet of thingsAudio / Video
Log Files
Text/Image
Social Sentiment
Data Market Feeds
eGov Feeds
Weather
Wikis / Blogs
Click StreamSensors / RFID / Devices
Spatial &
GPS Coordinates
WEB 2.0Mobile
Advertising CollaborationeCommerce
Digital Marketing
Search Marketing
Web Logs
Recommendations
ERP / CRM
Sales Pipeline
Payables
Payroll
Inventory
Contacts
Deal Tracking
Terabytes
(10E12)
Gigabytes
(10E9)
Exabytes
(10E18)
Petabytes
(10E15)
Variety, variability & Velocity
Vo
lum
e
1980
190,000$2010
0.07$
1990
9,000$2000
15$Storage/GB
ERP / CRM WEB 2.0Internet of
things
![Page 8: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/8.jpg)
![Page 9: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/9.jpg)
![Page 10: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/10.jpg)
![Page 11: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/11.jpg)
![Page 12: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/12.jpg)
What’s the
social sentiment
for my brand or
products ?
How do I
optimize my fleet
based on weather
and traffic
patterns?
How do I
better
predict future
outcomes?
![Page 13: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/13.jpg)
Big Data, BIG OPPORTUNITY
49% CEOs and CIOs are planning big data projects
Software Growth
1.8
2.5
3.4
4.6
0
1
2
3
4
5
2012 2013 2014 2015
Bil
lio
ns
$
34% compound
annual growth rate
Services Growth
2.7
3.9
5.1
6.5
0
2
4
6
8
2012 2013 2014 2015
Bil
lio
ns
$
39% compound
annual growth rate
1. McKinsey&Company, McKinsey Global Survey Results, Minding Your Digital Business, 2012
2. IDC Market Analysis, Worldwide Big Data Technology and Services 2012–2015 Forecast , 2012
![Page 14: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/14.jpg)
![Page 15: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/15.jpg)
![Page 16: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/16.jpg)
![Page 17: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/17.jpg)
![Page 18: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/18.jpg)
![Page 19: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/19.jpg)
Block #1
Block #3
Block #2
![Page 20: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/20.jpg)
Block #1
Block #3
Block #2
Block #1
Block #2
Block #3
![Page 21: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/21.jpg)
![Page 22: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/22.jpg)
![Page 23: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/23.jpg)
![Page 24: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/24.jpg)
Symbol Date High Low
AET 2009-09-21 30.49 31.09
AET 2009-09-18 31.01 31.44
…
MRK 1988-11-25 55.00 55.25
![Page 25: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/25.jpg)
Key Value
AET 0.61
AET 0.33
…
MRK 0.25
![Page 26: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/26.jpg)
Key Value
AET 13.75
AXP 15.12
…
MRK 29.00
![Page 27: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/27.jpg)
![Page 28: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/28.jpg)
Batch Processing Interactive analysis Stream processing
Query runtime Minutes to hours Milliseconds to minutes Never-ending
Data volume TBs to PBs GBs to PBs Continuous stream
Programming model MapReduce Queries DAG
Users Developers Analysts and developers Developers
Originating project Google MapReduce Google Dremel Twitter Storm
Open source project Hadoop / Spark Hive*/ Drill / Shark /Impala
Hbase
Storm / Apache S4 /Kafka
![Page 29: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/29.jpg)
Hive, Pig, Mahout, Cascading, Scalding, Scoobi, Pegasus…
C#, F# Map/Reduce, LINQ to Hive, .NET management
clients
JavaScript Map/Reduce, Browser hosted console, Node.js
management clients
PowerShell, Cross Platform CLI tools
![Page 30: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/30.jpg)
![Page 31: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/31.jpg)
![Page 32: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/32.jpg)
![Page 33: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/33.jpg)
![Page 34: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/34.jpg)
![Page 35: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/35.jpg)
![Page 36: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/36.jpg)
https://www.windowsazure.com/en-us/develop/net/how-to-guides/hadoop/
http://blogs.msdn.com/hpctrekker/
https://github.com/wenming/BigDataSamples
![Page 37: PowerPoint Presentationdownload.microsoft.com/documents/hk/technet/techdays2013...Batch Processing Interactive analysis Stream processing Query runtime Minutes to hours Milliseconds](https://reader030.vdocuments.mx/reader030/viewer/2022040921/5e98f57598354f1645253388/html5/thumbnails/37.jpg)