events: distributed programming made easy
TRANSCRIPT
EVENTS: DISTRIBUTED PROGRAMMING MADE EASY
STORM + F#
Storm + F#
ABOUT ME
Eugene TolmachevPrincipal Architect @ ProlucidDeveloping software for over 20 years Open source contributor:• FsStorm maintainer
• Bitlore author
• FsUnit
• Dapper
• MassTransit
• Nowin
Storm + F#
SCALE OUT & AVAILABILITY
UI Backend DB
A = Aui Abe Adb
Storm + F#
DISTRIBUTED COMPUTING
DRPC/Actors• CORBA/DCOM• Web-services• AKKA• Orleans
Data freshness
ESP/CEP• Storm• Spark
Batch processing• Custom ETL• Hadoop• mbrace• Spark
Storm + F#
ASK VS TELL
Model
Price
Risk Date
getRisk(date)risk
date
getModel(date) model
Stack: … getPrice getRisk getModel …
• Easy to start with• Composed at design-time• Fault handling: on the caller• Challenging to cache• Fragile integration
Storm + F#
EVENTS (FIRE-N-FORGET)
Implemented via Store-and-forward messagingRepresent a summary of a changeSelf-contained*Time-relevant, not time-sensitiveUniquely identifiable
• Tracing/correlation• Idempotency
Storm + F#
ASK VS TELL
• Upfront setup• Can compose at runtime• Fault handling:
Compensations• Caching – unnecessary• Can evolve independently• Embarrassingly parallel
Price
Risk
DateModel
modelUpdated(m,date)
riskUpdated(r,m,date)
newDay(date)
newDay(date)calculate Model
calculate Risk
calculate Price
generate Date
Multiple Stacks:
Storm + F#
MICRO-SERVICES
Services encapsulate• Business-logic• Can evolve independently as long as the surface API remains the same
Services control the context• Persistence• Security• Performance
Micro-services+ Do one thing only, do it well- Introduce latency- Deployment/management overhead
Storm + F#
THERMOSTAT ALERTSensor
is up
Sensor value
Settings updated
Write down raw data Evaluate against
the threshold Send notification
Read all sensors
Write down the settings
Write down the result
Storm + F#
APACHE STORMRuns on JREStorm
• Spouts: throttled• Streams: default and named• Tuples: anchored or not• Bolts: input – on demand/output – at will• Grouping (affinity): all, shuffle, fields, direct
Storm multilang protocol:• JSON over stdin/stdout• Components: a process per instance
Storm CLI, to start:• storm dev-zookeeper (runtime coordination)• storm nimbus (management/api)• storm supervisor (runs components)• storm ui (optionally, http://localhost:8080)
Storm + F#
THERMOSTAT TOPOLOGY
sensorStartedEvents[1]
sensorUpdateEvents[1]
settingsUpdateEvents[1]
sensorWriter[2]
sensorViewProjection[2]
sensorViewUpdatedNotifier[1]
settingsUpdatedNotifier[1]
settingsWriter[1]
sensorViewWriter[2]
allSensorsReader[1]
Storm + F#
FSSTORMRuntime• Implements multilang protocol• Lets you write components in F#• Additional implementations for unit-testing
DSL:• Define topology in F#• Validated
Scripts• Submit your topology to nimbus