![Page 1: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/1.jpg)
BigDataSpatial Analytics
Mansour Raad
![Page 2: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/2.jpg)
Story Time...
![Page 3: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/3.jpg)
is hereby granted to
to certify that he/she has completed to satisfaction
The CCDH Exam
Cloudera, Inc. 210 Portage Avenue Palo Alto, CA 94306 www.cloudera.com
___________________________ Date Granted
Test Date:
___________________________ Authorized Signature
Mansour Raad
March 2, 2012
Mar 09, 2012
is hereby granted to
to certify that he/she has completed to satisfaction
The CCDH Exam
Cloudera, Inc. 210 Portage Avenue Palo Alto, CA 94306 www.cloudera.com
___________________________ Date Granted
Test Date:
__ __________________________Authorized Signature
March 2, 2012
Mar 09, 2012
![Page 4: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/4.jpg)
![Page 5: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/5.jpg)
![Page 6: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/6.jpg)
Finally, a big nail...
![Page 7: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/7.jpg)
Input 1
![Page 8: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/8.jpg)
U.S.Demographic
Data
![Page 9: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/9.jpg)
Demographic Info
• Location
• Gender
• Race
• Income
• Age
![Page 10: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/10.jpg)
Input 2
![Page 11: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/11.jpg)
~1000 Locations
![Page 12: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/12.jpg)
Task...
![Page 13: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/13.jpg)
For Each LocationFor Each Demographic
50 Mile Heatmap
![Page 14: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/14.jpg)
![Page 15: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/15.jpg)
“Traditional Way”
• 14 Days Later
• 850GB Raster
![Page 16: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/16.jpg)
Gotta Be A Better Way !
![Page 17: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/17.jpg)
Hadoop
![Page 18: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/18.jpg)
$> cat input | map | sort | reduce > out
![Page 19: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/19.jpg)
![Page 20: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/20.jpg)
Advantage
• Parallelism
• Fast Input Stream
• Fast Computational Geometry
• Distributed Cache
![Page 21: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/21.jpg)
Vector / Raster
![Page 22: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/22.jpg)
Cooperative Processing
g.beginGradientFill(GradientType.RADIAL,[ 0xFF0000, 0x0000FF ], ...);g.drawRect(x, y, 200, 200);g.endFill();bitmapData.draw(shape, null, null, BlendMode.SCREEN, null, true);
![Page 23: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/23.jpg)
Where To Run 10 Nodes ?
![Page 24: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/24.jpg)
![Page 25: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/25.jpg)
![Page 26: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/26.jpg)
![Page 27: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/27.jpg)
~238 MB Vectorvs.
~850 GB Raster
![Page 28: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/28.jpg)
![Page 29: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/29.jpg)
Best Visualizer ?
![Page 30: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/30.jpg)
![Page 31: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/31.jpg)
What is Big Data ?
![Page 32: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/32.jpg)
Great Story Telling Tool !
Data Democratizer!Beyond Dashboard!Can have best ML, best model, best team, all useless if u cannot tell a story of results!
![Page 33: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/33.jpg)
What Is Big Data ?
(academic)
![Page 34: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/34.jpg)
Beyond Traditional Means !
Traditional Processing
Traditional Database
![Page 35: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/35.jpg)
•Too Big
•Too Fast
•Unstructured
![Page 36: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/36.jpg)
Forcing new ways of thinking !
![Page 37: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/37.jpg)
Big Data Sources...
Catch all wordsjust like “Cloud” was 3 year ago !
![Page 38: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/38.jpg)
![Page 39: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/39.jpg)
WebLogs
![Page 40: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/40.jpg)
“Internet Of Things”
![Page 41: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/41.jpg)
Imagery
![Page 42: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/42.jpg)
Health Records
![Page 43: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/43.jpg)
VOLUME
VELOCITY VARIETY
![Page 44: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/44.jpg)
Volume
• Very Large Amount
• More Parameters
• Multi Node
• Storage
• Processing -Simple math is more effective with large parameters-Scalable storage-Program to data rather data to program
![Page 45: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/45.jpg)
Velocity
• Rate of digital flow
• Streaming
• Event Processing
• Feedback Loop
• Recommendations - Clicks, locations- Mobile / Smartphones- Last 5 min snapshot of traffic is no good when crossing the street- CERN
![Page 46: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/46.jpg)
Velocity Engines
• IBM InfoSphere Streams
• Twitter Storm
• Apache S4
![Page 47: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/47.jpg)
Variety
• Unstructured
• Incomplete
• Semantically Different
Data is messy
![Page 48: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/48.jpg)
Storage Variety
• NoSQL
• Columnar (HBase)
• Key/Value (Redis)
• Document (MongoDB)
• Graph (Neo4J)
![Page 49: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/49.jpg)
Hadoop
![Page 50: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/50.jpg)
HDFS
• Multi-TB Storage
• Inexpensive Nodes
• Fault Tolerant
• Concurrent Reading
• Brings Programs To Data
![Page 51: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/51.jpg)
MapReduce
• Software Framework
• Parallel Processing
• Jobs Executed on HDFS
• Java / Python / C++
• Spatial Libraries
![Page 52: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/52.jpg)
MapReduce Job
input | map | sort | reduce | output
Java Jars packaged and sent to data nodes for execution
![Page 53: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/53.jpg)
Apache Hive
![Page 54: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/54.jpg)
“SQL”
MapReduce Job
![Page 55: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/55.jpg)
HDFSCSVTSV
JSONBINARY
MapReduce
hive> select * from cities where country=‘lebanon’;
![Page 56: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/56.jpg)
Spatial Storage
• CSV,TSV Lat,Lon
• Esri JSON format
• {geometry:{x:-123,y:45},attributes:{}}
• Custom
![Page 57: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/57.jpg)
What About Spatial ?
![Page 58: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/58.jpg)
User Defined Functions
• select tolower(“ESRI”);
• select * from mytable where cos(rad) < 0.1;
![Page 59: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/59.jpg)
Spatial UDF !
![Page 60: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/60.jpg)
select * from citieswhere near(x,y,-84.2,39.4);
![Page 61: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/61.jpg)
select * from citieswhere contains(x,y,’#mypolys’);
![Page 62: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/62.jpg)
PythonGeoProcessing
![Page 63: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/63.jpg)
HDFSRDBMS
“small data” “big data”
HadoopTools
ArcMapCatalog
![Page 64: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/64.jpg)
Demo Time
![Page 65: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/65.jpg)
The “Zoo”
• Pig - high level language for hadoop
• HBase - real/time random access to hdfs
• Flume - streaming data flow
• Mahout - machine learning
• Zookeeper - distributed state management
![Page 66: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/66.jpg)
Processing Evolution
• Transactional - Batch
• Operational - Dashboard
• Analytical - Exploratory
• Intelligent - Real/Time, predictive
Fixed Schema
Variable Schema
![Page 67: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/67.jpg)
“[T]here are known knowns; there are things we know that we know.There are known unknowns; that is to say there are things that, we now know we don't know.But there are also unknown unknowns – there are things we do not know we don't know.”
—United States Secretary of Defense, Donald Rumsfeld
![Page 69: BigData Spatial Analytics - Amazon S3...Hadoop Tools ArcMap Catalog Demo Time The “Zoo” • Pig - high level language for hadoop • HBase - real/time random access to hdfs •](https://reader031.vdocuments.mx/reader031/viewer/2022011912/5fa5ddc7188be76ee470088d/html5/thumbnails/69.jpg)
Date Event Location
March 21, 2013Esri DC Meet Up – Big Data & Location Analytics Washington, DC
April 18, 2013 Esri DC Meet Up Washington, DC
March 23–26, 2013 Esri Partner Conference Palm Springs, CA
March 25–28, 2013 Esri Developer Summit Palm Springs, CA
July 6–9, 2013 Esri National Security Summit San Diego, CA
July 8–12, 2013 Esri International User Conference San Diego, CA
Upcoming Events