Download - Geo-Trending Example
Transcript
![Page 1: Geo-Trending Example](https://reader038.vdocuments.mx/reader038/viewer/2022100801/589fe6701a28abf3238b5d55/html5/thumbnails/1.jpg)
Trending Topics by Geo-Location
Insight Data Engineering Tigran Antonyan
New York
![Page 2: Geo-Trending Example](https://reader038.vdocuments.mx/reader038/viewer/2022100801/589fe6701a28abf3238b5d55/html5/thumbnails/2.jpg)
Demo!!! API/Web Form Works!
• Only the a simple API/Form works so far ‣ via Flask
• http://trendinghashtags.net:5000
![Page 3: Geo-Trending Example](https://reader038.vdocuments.mx/reader038/viewer/2022100801/589fe6701a28abf3238b5d55/html5/thumbnails/3.jpg)
Data Transformation
Map ReduceFilter Map Map Reduce
![Page 4: Geo-Trending Example](https://reader038.vdocuments.mx/reader038/viewer/2022100801/589fe6701a28abf3238b5d55/html5/thumbnails/4.jpg)
Data Flow / PipelineMessageBroker
Real-TimeStreaming
DBNo-SQL
UIWeb-based
Camus
Batch Processing
Some work remains
❌
❌
❌
![Page 5: Geo-Trending Example](https://reader038.vdocuments.mx/reader038/viewer/2022100801/589fe6701a28abf3238b5d55/html5/thumbnails/5.jpg)
Cluster Setup
HDFS Namenode
Camus (cron) / Kafka Consumer
Spark Master
Kafka Broker
HDFS Datanode #1
Stream Reader / Kafka Producer
Kafka BrokerSpark
HDFS Datanode #2
Kafka BrokerSpark
HDFS Datanode #2
Kafka BrokerSpark
Cassandra Seed
Cassandra
Cassandra
m4.large nodes m3.medium nodes
![Page 6: Geo-Trending Example](https://reader038.vdocuments.mx/reader038/viewer/2022100801/589fe6701a28abf3238b5d55/html5/thumbnails/6.jpg)
About MeCamping
• BS in CS at FAU
• Software Engineer
‣ Citrix
‣ UCONN VoTeR Center