Building End to End Streaming Application on Spark

Streaming application development journey

https://github.com/Shasidhar/sensoranalytics

● Shashidhar E S

● Big data consultant and trainer at datamantra.io

● www.shashidhare.com

Agenda● Problem Statement● Spark streaming● Stage 1 : File Streams● Stage 2 : Kafka as input source (Introduction to Kafka)● Stage 3 : Casandra as Output Store (Introduction to Cassandra)● Stage 4 : Flume as data collection engine (Introduction to Flume)● How to test streaming code?● Next steps

Earlier SystemBusiness model

● Providers of Wi-Fi hot spot devices in public spaces● Ability to collect data from these devices and analyse

Existing System

● Collect data and process in daily batches to generate the required results

Existing System

Server

Server

Server

Server

Central directory Splunk Downstream

Systems

Need for real time engine● Lot of failures in User login● Need to analyse why there is a drop in user logins● Ability to analyse the data in real time rather than daily

batches● As the company is growing Splunk was not scaling as it is

not meant for horizontal scaling

New system requirement● Able to collect and process large amount of data ● Ability to store results in persistent storage● A reporting mechanism to view the insights obtained from

the analysis● Need to see the results in real time● In a simple term, we can call it as a real time monitoring

system

Why Spark Streaming ? ● Easy to port batch system to streaming engine in Spark● Spark streaming can handle large amounts of data and it

is very fast● Best choice for near real time systems● Futuristic views

○ Ability to ingest data from many sources○ Good support for downstream stores like NoSQL○ And lot more

Spark Streaming Architecture

Server Source directory

Spark Streaming

engine

Output directory

View in Zeppelin

Data formatLog Data with the following format

● Timestamp● Country● State● City● SensorStatus

Required Results● Country Wise Stats

○ Hourly,Weekly and Monthly view of total count of records captured countrywise.

● State Wise Stats○ Hourly,Weekly and Monthly view of total count of records captured

statewise.

● City Wise Stats○ Hourly,Weekly and Monthly view of total count of records captured city

wise with respect to sensor status

Data Analytics - Phase 1 ● Receive data from servers● Store the input data into files● Use file as input and output● Process the data , generate

required statistics● Store results into output files

Spark Streaming engine

Input files (Directory)

Output files (Directory)

Spark streaming introduction

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams

Micro batch

● Spark streaming is a fast batch processing system● Spark streaming collects stream data into small batchand runs batch processing on it● Batch can be as small as 1s to as big as multiple hours● Spark job creation and execution overhead is so low it

can do all that under a sec● These batches are called as DStreams

Apache Zeppelin● Web based notebook that allows interactive data analysis● It allows

○ Data ingestion○ Data Discovery○ Data Analytics○ Data Visualization and collaboration

● Built-in Spark integration

Data Model● 4 models

○ SensorRecord - To read input records○ CountryWiseStats - Store country wise aggregations○ StateWiseStats - Store state wise aggregations○ CityWiseStats - Store city wise aggregations

Phase 1 - Hands On

Git branch : Master

Problems with Phase 1● Input and output is a file● Cannot detect new records / new data as and when it is

received● File causes Low latency in system

Solution : Replace Input file source with Apache kafka

Data Analytics - Phase 2 ● Receive data from servers● Store the input data in Kafka● Use kafka as input ● Process the data , generate

required statistics● Store results into output files

Spark Streaming engine

Kafka

Output files (Directory)

Apache Kafka● High throughput publish subscribe based messaging

system● Distributed, partitioned and replicated commit log● Messages are persistent in system as Topics● Uses Zookeeper for cluster management● Written in scala, but supports many client API’s - Java,

Ruby, Python etc● Developed by LinkedIn

High Level Architecture

Terminology● Topics : Is where messages are maintained and

partitioned● Producers : Processes which produces messages to

Topic● Consumers: Processes which subscribes to topic and

read messages● Brokers: Every server which is part of kafka cluster

Anatomy of Kafka Topic

Spark Streaming - Kafka● Two ways to fetch data from kafka to spark

○ Receiver approach■ Data is stored in receivers■ Kafka topic partitions does not correlate with RDDs■ Enable WAL for zero data loss■ To increase input speed create multiple receivers

Spark Streaming - Kafka cont○ Receiver less approach

■ No data is stored in receivers■ Exact same partitioning in maintained in Spark RDDs as in

Kafka topics■ No WAL is needed as data is already in kafka we can fetch

older data on receiver crash■ More kafka partitions increases the data fetching speed

Phase 2 - Hands On

Git branch : Kafka

Problems with Phase 2● Output is still a file● Always full file scan is needed to retrieve, no lookups● Querying results is cumbersome● Nosql Database is the better option

Solution : Replace Output file with Cassandra

Data Analytics - Phase 3

Spark Streaming engine

Kafka

Cassandra

● Receive data from servers● Store the input data in Kafka● Use kafka as input● Process the data , generate

required statistics● Store results into cassandra

What is Cassandra “Apache Cassandra is an open source, distributed, decentralized, elastically scalable, highly available, fault-tolerant, tunable consistency, column-oriented database”

“Daughter of Dynamo and Big Table”

Key Components and Features

● Distributed● System keyspace ● Peer to peer - No SPOF● Read and write to any node ● Operational simplicity● Gossip and Failure Detection

Cassandra daemon

cassandra(CLI)

Language driversJDBC Drivers

Memtable SS tablesCommit Log

Overall Architecture

Spark Cassandra Connector● Loads data from cassandra to spark and vice versa● Handles type conversions● Maps tables to spark RDDs● Support all cassandra data types, collections and UDTs● Spark-Sql support● Supports for Spark SQLs predicate push

Phase 3 - Hands On

Git branch : Cassandra

Problems with Phase 3● Servers cannot push directly to Kafka● There is an intervention to push data● Need for automated way to push data

Solution : Add Flume as a data collection agent

Data Analytics - Phase 4 ● Receive data from Server● Stream data into kafka through

flume● Store the input data in Kafka● Use kafka as input● Process the data , generate

required statistics● Store results into cassandra

Spark Streaming engine

Kafka

Cassandra

Flume

Apache Flume● Distributed data collection service● Solution for data collection of all formats● Initially designed to transfer log data into HDFS frequently

and reliably● It is horizontally scalable● Configurable routing

Flume ArchitectureComponents

○ Event○ Source ○ Sink○ Channel○ Agent

Flume Configuration● Define Source, Sink and Channel names● Configure Source● Configure Sink● Configure Channel● Bind Source and Sink to Channel

Phase 4 - Hands On

Git branch : Flume

Data Analytics - Re Design● Why we want to re design/ re structure ?● What we want to test ?● How to test Streaming applications● Hack a bit on Spark Manual Clock● Use scala-test for unit testing● Bring up abstractions to decouple the code● Write some tests

Manual Clock

● A clock whose time can be set and modified● Its notified time will not change as time elapses● Only callers have control over it● Specially used for testing

Phase 5 - Hands On

Git branch : unittest

Next steps● Use better serialization frameworks like Avro● Enable Checkpointing● Integrate kafka monitoring tools● Adding support for multiple kafka topics● Write more tests for all functionality