high volume streaming data: how amazon web services is changing our approach
TRANSCRIPT
How Amazon Web Services is Changing Our Approach
Presented by: Michael Krouze, CTO & VP Analytics, Charter Solutions, Inc.
High Volume Streaming Data
2
Act I
1990
3
4
Improve the quality of the manufacturing process
Build something usable with minimal education
Distributed system
High-visualization
Intuitive
5
Data Collection Server
Archive Server
Visualization Station
Sensors
Data Collection Server
Process Control
Data Collection Server
Manual Entry
Visualization Station
Visualization Station
Client 1
6
Visualization Software
GDBM Client
Client 2
Visualization Software
GDBM Client
Source
Integration Code
GDBM Client
ArchiveServer
Observations7
u Met the needs of the organizationu Was a cool, cutting edge projectu Took 5 internal staff and 3 external consultants years to implementu Had measurable positive impact on product quality
u Very customu Lots of manual labor in scaling upu Costly
Act II
2011
8
9
Support a new set of services around HVAC management
Improve customer satisfaction
Build better products
High availability
Linear scalability
Fast
10
Amazon EC2
AmazonSQS
AmazonS3
InstanceQueue
Bucket
DB on Instance
Amazon VPC
Internet Gateway
Router
Elastic Load Balancing
11
Bucket
Queue
Vertica Database
3..n Instances
Catching Mitt
Custom Java App
2..n Instances
Data PrepCustom
Java App2..n
Instances
Viz Server
SAS Server
Real Time AnalyticsCustom
Java/R App
AWS cloud
BuildingDevice
user
user
Corporate Data Center
Observations12
u 2 part-time resources took 2 months to implement initial versionu Increased performance by over 60xu Cost improvements of over 50%u Productivity increase of over 40%u Faster time from data collection to visualizationu Scales linearly and quicklyu Highly failure resilient
u Must maintain capacity for bursts of datau Delay of up to 5 minutes to scale
Act III
2014
13
14
Provide insight into how users engage with the application
Provide insight into content usage for ordering and recommendation models
Low cost of entry
Fast
Can scale quickly, easily, and in small increments
Can handle “bursty” traffic
15
Amazon Kinesis
AWSLambda
AmazonS3
Amazon Redshift
Amazon CloudWatch
Amazon Kinesis Streams
Bucket AlarmDense Storage Node
AWS KMS Amazon VPC
Internet Gateway
Router
16
Core Application
EventStream
Data Processing
Data Store
Long-TermStorage
Analytics ClientMobile
Browser
AWS cloud
Observations17
u 1 person implemented in 2-3 weeksu Very low cost for low data ratesu Scales in very small incrementsu Scales fastu Handles small and large bursts of data easilyu Minimal held capacity – primarily pay when processing data
18
A Note on Scale19
Rate per Second Rate per Month Rate per Year
1 KB 2.6 GB 31 GB
A Note on Scale20
Rate per Second Rate per Month Rate per Year
1 KB 2.6 GB 31 GB10 KB 26 GB 310 GB100 KB 260 GB 3.1 TB1 MB 2.6 TB 31 TB10 MB 26 TB 310 TB100 MB 260 TB 3.1 PB
21
22
23
24
25
Act IV
Today and Beyond
26
27
28
AWS Import/Export Snowball
AmazonDynamoDB
Amazon Kinesis
Amazon Machine Learning
Amazon QuickSight
AWS IoTAmazon ElasticsearchService
import/export
Amazon Kinesis Firehose
Amazon Kinesis
Analytics
IoT HTTPprotocol
IoTcertificate
IoTrule
29
Final Words30
u IoT will drive the future of streaming data analyticsu While AWS has a strong lead, there are competitors both in the general cloud services arena
and individual nichesu Batched data for analysis will decrease over time – giving preference to streaming data
u Even for transactional systems
Thank you!
u Contact me:u [email protected] @mjkrouze
u Resources:u www.chartersolutions.comu github.com/awslabs/amazon-redshift-utilsu AWS YouTube channelu AWS on SlideShare