future of apache storm

Future of Apache StormHadoop Summit 2016, Melbourne

About usArun Iyer, Hortonworks

Apache Storm Committer/PMCarunm@apache.org

Satish Duggana, HortonworksApache Storm Committer/PMCsatishd@apache.org

Apache Storm 1.0Maturity and Improved Performance

Release Date: April 12, 2016

Distributed Cache API

Topology resources– Dictionaries, ML Models, Geolocation Data, etc.

Typically packaged in topology jar– Fine for small files– Large files negatively impact topology startup time– Immutable: Changes require repackaging and deployment

Allows sharing of files (BLOBs) among topologies

Files can be updated from the command line

Allows for files from several KB to several GB in size

Files can change over the lifetime of the topology

Allows for compression (e.g. zip, tar, gzip)

Two implementations: LocalFsBlobStore and HdfsBlobStore Local implementation supports Replication Factor (not needed for HDFS-backed

implementation) Both support ACLs

Creating a blob:

– storm blobstore create --file dict.txt --acl o::rwa --repl-fctr 2 key1

Making it available to a topology:

– storm jar topo.jar my.topo.Class test_topo -c topology.blobstore.map=‘{"key1":{"localname":"dict.txt", "uncompress":"false"}}'

Highly Available Nimbus

Nimbus

Nimbus HA

ZooKeeper

Supervisor

Worker*

Supervisor

Worker*

Supervisor

Worker*

Nimbus

Leader

Nimbus

Nimbus HA

ZooKeeper

Supervisor

Worker*

Supervisor

Worker*

Supervisor

Worker*

Nimbus

Leader

Nimbus

Nimbus HA

ZooKeeper

Supervisor

Worker*

Supervisor

Worker*

Supervisor

Worker*

Leader election

Nimbus

Leader

Nimbus

Nimbus HA

ZooKeeper

Supervisor

Worker*

Supervisor

Worker*

Supervisor

Worker*

Nimbus

Leader

Nimbus

Nimbus HA

ZooKeeper

Supervisor

Worker*

Supervisor

Worker*

Supervisor

Worker*

Nimbus HA

Increase overall availability of Nimbus Nimbus hosts can join/leave at any time Leverages Distributed Cache API Topology JAR, Config, and Serialized Topology uploaded to Distributed Cache Replication guarantees availability of all files

PacemakerHeartbeat Server

Pacemaker

Replaces Zookeeper for Heartbeats In-Memory key-value store Allows Scaling to 2k-3k+ Nodes Secure: Kerberos/Digest Authentication

Pacemaker

Compared to Zookeeper– Less Memory/CPU– No Disk– No overhead of maintaining consistency

Automatic Back Pressure

In previous Storm versions, the only way to throttle topologies was to enable ACKing and set topology.spout.max.pending.

If you don’t require at-least-once guarantees, this imposed a significant performance penalty.

High/Low Watermarks (expressed as % of task’s buffer size) Back Pressure thread monitors task’s buffers If High Watermark reached, slow down Spouts If Low Watermark reached, stop throttling All Spouts Supported

Performance

Up to 16x faster throughputRealistically 3x -- Highly dependent on use case and fault tolerance settings

> 60% Latency Reduction

Resource Aware Scheduler(RAS)

Resource Aware Scheduler

Specify the resource requirements (Memory/CPU) for individual topology components (Spouts/Bolts)

Memory: On-Heap / Off-Heap (if off-heap is used) CPU: Point system based on number of cores Resource requirements are per component instance (parallelism

matters)

CPU and Memory availability described in storm.yaml on each supervisor node. E.g.:

supervisor.memory.capacity.mb: 3072.0supervisor.cpu.capacity: 400.0

Convention for CPU capacity is to use 100 for each CPU core

SpoutDeclarer spout = builder.setSpout("sp1", new TestSpout(), 10);//set cpu requirementspout.setCPULoad(20);//set onheap and offheap memory requirementspout.setMemoryLoad(64, 16);

BoltDeclarer bolt1 = builder.setBolt("b1", new MyBolt(), 3).shuffleGrouping("sp1");//sets cpu requirement. Not neccessary to set both CPU and memory.//For requirements not set, a default value will be usedbolt1.setCPULoad(15);

BoltDeclarer bolt2 = builder.setBolt("b2", new MyBolt(), 2).shuffleGrouping("b1");bolt2.setMemoryLoad(100);

Native windowing support

Fundamental abstraction for processing continuous streams of events

Breaks the stream of events into sub-sets

The sub-sets can then be aggregated, joined or analyzed.

Why is it important?

Trending items– The top trending search queries in the last day

– Top tweets in the last 1 hour

Complex event processing– Windowing is fundamental to gain meaningful insights from a stream of events

– E.g. If same credit card is used from geographically distributed locations within the last one hour, it could be a fraud.

Some use cases

Sliding Window

time0 5 10 15

Tumbling Window

Windows do not overlap

Windows can overlap

Native windowing supportpublic class WindowMaxBolt extends BaseWindowedBolt { private OutputCollector collector; public void prepare(Map conf, TopologyContext context, OutputCollector collector) { this.collector = collector; }

public void execute(TupleWindow inputWindow) { int max = Integer.MIN_VALUE; for (Tuple tuple: inputWindow.get()) { max = Math.Max(max, tuple.getIntegerByField(”val")); } collector.emit(new Values(max)); }

public static void main(String[] args) { // ... builder.setBolt("slidingMax", new WindowMaxBolt().withWindow(Duration.minutes(10), Duration.minutes(2)));

// submit topology StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology()); }}

triggered when the window activates based on the window configuration

the window configuration

Native windowing supportTrident

topology.newStream("spout", spout) .window( SlidingDurationWindow.of(Duration.minutes(10), Duration.minutes(2)), // window configuration new InMemoryWindowsStoreFactory(), // window store factory new Fields("val"), // input field new Min("val"), // aggregate function new Fields("max") // output field );

Processing time Event time

– e.g. Processing events out of a log file based on when the event actually happened– Implemented based on watermarks

Out of order and late events– e.g. Mobile gaming data where the user goes offline and data arrives after sometime

Windowing considerations

6:026:01 6:05 6:08

7:017:00

6:10 6:12

Processing time

6:09 6:126:08

watermark watermark

late event

State ManagementStateful Bolts with Automatic Checkpointing

State Management

Allows storm bolts to store and retrieve the state of its computation

The framework automatically and periodically snapshots the state of the bolts

In-memory and Redis based state stores

Implementation based on– Asynchronous Barrier Snapshotting (ABS) algorithm [1]– Chandy-Lamport Algorithm [2]

1. http://arxiv.org/abs/1506.086032. http://research.microsoft.com/en-us/um/people/lamport/pubs/chandy.pdf

State Managementpublic class WordCountBolt extends BaseStatefulBolt<KeyValueState> {

private KeyValueState wordCounts; private OutputCollector collector;

public void prepare(Map conf, TopologyContext context, OutputCollector collector) { this.collector = collector; }

public void initState(KeyValueState state) { this.wordCounts = state; }

public void execute(Tuple tuple) { String word = tuple.getString(0); Integer count = (Integer) wordCounts.get(word, 0); count++; wordCounts.put(word, count); collector.emit(new Values(word, count)); }}

Initialize state

Update state

State Management

Checkpointing/Snapshotting: What you see

Spout Stateful Bolt1 Stateful Bolt2Bolt1

execute/update state execute/update stateexecute

State Management

Checkpointing/Snapshotting: What you get

Checkpoint Spout

Stateful Bolt1 Stateful Bolt2Bolt1

State Store

ACKACK

chkpt chkpt

Streaming SQL(currently Beta/WIP)

Streaming SQL

SQL is a well adopted standard– available in several projects like Drill, Hive, Phoenix, Spark

Faster development cycle Opportunity to unify batch and stream data analytics

Benefits

SELECT word, COUNT(*) AS totalFROM wordsGROUP BY word

Streaming SQL

Leverages Apache calcite Compiles SQL statements into Trident topologies Usage:

– $ bin/storm sql <sql-file> <topo-name>

Storm implementation

Streaming SQL

Example

CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders’

CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) LOCATION 'kafka://localhost:2181/brokers?topic=large_orders’

INSERT INTO LARGE_ORDERSSELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM ORDERS WHERE UNIT_PRICE * QUANTITY > 50

Streaming SQL

Storm SQL workflow

Streaming SQL

Storm SQL workflow

Storm Usability ImprovementsEnhanced Debugging and Monitoring of Topologies

Storm usability improvements

./bin/storm set_log_level [topology name]-l [logger name]=[LEVEL]:[TIMEOUT]

Dynamic Log Levels

Storm CLI:

Storm UI:

No more debug bolts or Trident functions Click on the “Events” link to view the sample log.

Tuple Sampling

Distributed Log search– Search across all topology logs, even archived (ZIP) logs.

Dynamic worker profiling– Heap dumps, JStack output, Profiler recordings

Restart workers from UI Supervisor health checks

Integrations

New Cassandra Solr Elastic Search MQTT

Improvements Kafka HDFS spout Hbase Hive

What’s next for Storm?2.0 and Beyond

Clojure to JavaBroadening the contributor base

Alibaba JStorm Contribution

Apache Beam (incubating) Integration Unified API for dealing with

bounded/unbounded data sources (i.e. batch/streaming)

One API. Multiple implementations (execution engines). Called “Runners” in Beamspeak.

Apache Storm 2.0 and beyond

Windowing enhancements Unified Stream API Revamped UI Metrics improvements

and many more…

Q&AThank You

future of apache storm

Technology

hadoop summit europe 2014: apache storm architecture

streaming apache storm · 2020-01-06 · 3 5 apache storm 6...

apache storm - introduction au traitement temps-réel avec...

resource aware scheduling in apache storm

apache storm 0.9 basic training - verisign

low latency scalable web crawling on apache storm

learning stream processing with apache storm

hortonworks data platform for hdinsight · • apache pig...

apache storm

real time and reliable processing with apache storm

punch clock for debugging apache storm

real-time streaming with apache spark streaming and apache...

slide #2: setup apache storm

quantitative analysis of apache storm applications: the...

apache storm basics

introduction to apache nifi and storm

distributed and fault tolerant realtime computation with...

the future of apache storm

apache storm internals

scaling apache storm (hadoop summit 2015)