using aws to build a scalable machine data analytics service
TRANSCRIPT
![Page 1: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/1.jpg)
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Using AWS To Build
A Scalable Machine Data Analytics Service
Christian Beedgen
November 13, 2013
![Page 2: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/2.jpg)
Who Am I
• Co-Founder & CTO, Sumo Logic since 2010– Cloud-based Machine Data Analytics Service
– Applications, Operations, Security
• Server guy, Chief Architect, ArcSight, 2001-2009– Major SIEM player in the enterprise space
– Log Management for security & compliance
![Page 3: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/3.jpg)
Everything You Know Is Wrong
![Page 4: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/4.jpg)
Everything You Know Is Wrong
![Page 5: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/5.jpg)
Agenda
• Introduction To Logs & Logging
• Why We Are Building This Service
• Architecture Of The Service
• Deployment Automation
• Loosely Coupled Components
• Lessons Learned
• Cost & Business Value
![Page 6: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/6.jpg)
Introduction To Logs & Logging
![Page 7: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/7.jpg)
What Is Machine Data?
• Actually, Machine Generated Data
Curt Monash:
“Data that was produced
entirely by machines OR
data that is more about
observing humans than
recording their choices.”
Daniel Abadi:
"Machine-generated data is
data that is generated as a
result of a decision of an
independent computational
agent or a measurement of
an event that is not caused
by a human action."
![Page 8: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/8.jpg)
Examples Of Machine Data
• Computer, network, and other equipment logs
• Satellite and similar telemetry (espionage or science)
• Location data, RFID chip readings, GPS system output
• Temperature and other environmental sensor readings
• Sensor readings from factories, pipelines, etc.
• Output from many kinds of medical devices
![Page 9: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/9.jpg)
What Are Logs?
• Logs are a kind of Machine Data
• Time-stamped bits and pieces of text
• Whispers & utterances of your infrastructure
• Written to disk to a log file by applications
• Sent over the network by devices
![Page 10: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/10.jpg)
![Page 11: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/11.jpg)
A Wealth Of Information
• Like Twitter for your infrastructure
• Machine data analytics…
• …is sentiment analysis for machines
• Free data of tremendous value
• Don’t forget to manage and analyze it
![Page 12: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/12.jpg)
Or Else…
![Page 13: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/13.jpg)
Anatomy Of A Log
![Page 14: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/14.jpg)
Anatomy Of A Log
• Timestamp with time zone!
![Page 15: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/15.jpg)
Anatomy Of A Log
• Timestamp with time zone!
• Log level
![Page 16: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/16.jpg)
Anatomy Of A Log
• Timestamp with time zone!
• Log level
• Host ID & module name (process/service)
![Page 17: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/17.jpg)
Anatomy Of A Log
• Timestamp with time zone!
• Log level
• Host ID & module name (process/service)
• Code location or class
![Page 18: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/18.jpg)
Anatomy Of A Log
• Timestamp with time zone!
• Log level
• Host ID & module name (process/service)
• Code location or class
• Authentication context
![Page 19: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/19.jpg)
Anatomy Of A Log
• Timestamp with time zone!
• Log level
• Host ID & module name (process/service)
• Code location or class
• Authentication context
• Key-value pairs
![Page 20: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/20.jpg)
Use Cases
• Availability & Performance– Prevent downtime by proactive analytics, alerting
– Reduce MTTR by having all required data at your fingertips
• Application Release– Derive metrics from development and staging systems pre-deploy
– Baseline and compare after post-deploy quickly shows errors
• Security & Compliance– Compliance starts with having all security related logs in one place
– Analytics across all data facilitates detecting breaches and problems
![Page 21: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/21.jpg)
Use Case Customer Examples Metric
Security &
Compliance
Apigee reduced compliance
audit costs by ~50%
Availability and
Performance
Ink saves nearly $500K
annually
Application
Release
Intaact reduced errors
by 4X
Customer Metrics
![Page 22: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/22.jpg)
Machine Data Is Big Data
• Volume– Machine Data is voluminous and will continue to grow
– Our own application creates 1TB/logs per week easily
• Velocity– Machine Data occurs in real-time, and it is time-stamped
– Needs to be processed in real-time as well
• Variety– Machine Data is unstructured, or poly-structured at best
– Some standard schema, but sure enough not for you applications
![Page 23: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/23.jpg)
Why We Are Building This Service
![Page 24: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/24.jpg)
We Need To Evolve
![Page 25: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/25.jpg)
We Need To Evolve
![Page 26: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/26.jpg)
Legacy Products Fall Short
• Volume leads to scalability issues– Every Log Management system will fail – I have seen it
– Why should you bother with scaling yet one more system?
• Velocity challenges processing pipelines– What good are dashboards if they are not real-time?
– Streaming query engines are absolute must
• Variety isn’t being embraced– All data should be allowed into the system
– No vendor will ever know your application’s log schema
![Page 27: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/27.jpg)
AWS Enables Innovation
• Attending Werner’s talk at Stanford in 2008
• First parking lot discussion
• This can apply to our space!
• Datacenter as API
• Massive power up to scraggly devs
![Page 28: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/28.jpg)
AWS Enables Sumo Logic
• Entering an existing market – Existing & established competition, some of it huge
– Catch up & differentiate at the same time
• A Big Data service– Scaling on premise is hard and leaves the hard part to the customer
– Now we build one single system to deal with all customers
• This data is important– Regulatory compliance is among the big drivers for collecting it
– HA & DR concerns all over the place Amazon S3
![Page 29: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/29.jpg)
Deployment Architecture - Before
![Page 30: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/30.jpg)
Deployment Architecture - After
![Page 31: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/31.jpg)
Architecture Of The System
![Page 32: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/32.jpg)
Development Approach
• Developed in Scala because we like it
• Many small cohesive modules, low coupling
• Maven-based build system
• Layers of modules combined into applications
• Different applications for different concerns
• Internal Service-Oriented Architecture
• Communication via documented protocols
![Page 33: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/33.jpg)
Basic Concerns
• Data ingestion
– Receiving data
– Raw storage
– Full-text indexing
• Data analysis
– Interactive analytics
– Scheduled queries
– Machine learning
– Continuous query
evaluation
![Page 34: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/34.jpg)
Concerns Map To Clusters
• A cluster is multiple instances of the same application
• Deployed on multiple Amazon EC2 instances
• Deployed across multiple availability zones
• Instances within a cluster are oblivious of each other
• Receive from upstream, talk to downstream
• Receive from message bus, or talk RPC
![Page 35: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/35.jpg)
Ingestion Path
Receiver Bus Index
Raw
CQ
S3
![Page 36: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/36.jpg)
Receiver
• HTTPS endpoint behind Elastic Load Balancing
• Decompress messages from Collector
• Extract timestamps from messages
• Aggregate messages per-customer into blocks
• Flush blocks to message bus
• Ack to Collector
• “Statelessly stateful”/”Statefully stateless”
Receiver
![Page 37: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/37.jpg)
Raw
• Receive message blocks from message bus
• Encrypt message blocks
• Different key for every day for every customer
• Flush encrypted message blocks to Amazon S3
• Copy blocks as CSV to customer’s Amazon S3 bucket
• Ack to message bus
• Fully stateless
Raw
![Page 38: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/38.jpg)
Index
• Receive message blocks from message bus
• Cache message block on disk and ack to message bus
• Add message blocks to Lucene indexes
• Deal with wildly varying timestamps
• Flush index shards to Amazon S3
• Update meta data database with index shard info
• Stateful
Index
![Page 39: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/39.jpg)
Continuous Query
• Receive message blocks from message bus
• Evaluate each message against all search expressions
• Push matching messages into respective pipelines
• Ack to message bus
• Flush results periodically for pickup by client
• Persist checkpoints periodically to Amazon S3
• Stateful, with checkpoint recovery
CQ
![Page 40: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/40.jpg)
Analytics Path
Query
Service
CQ
S3
![Page 41: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/41.jpg)
Query
• Fully distributed streaming query engine
• Materialize messages matching search expression
• Push messages through a pipeline of operators
• First stage – non-aggregation operators
• Second stage – aggregation operators
• Present both raw message results as well as aggregates
• Results update periodically for interactive UI experience
Query
![Page 42: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/42.jpg)
Deployment Automation
![Page 43: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/43.jpg)
Why Deployment Automation
• Add 1 part developers, 1 part Datacenter-as-API, stir…
• Aim for fully integrated continuous deployment
• Checkin unit test integration test deployment
• Jenkins automates it all – using AWS instances
• Deployment doesn’t mean production
• Nite Stag Long Prod deployments
• There are humans involved as well!
![Page 44: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/44.jpg)
Automation Enables Scale
• The goal is 100% - accept no less
• Why U need automation– Number of deployments grows (staging, per-developer)
– Number of AWS resources per deployment grows
– Number of operators/developers grows
– Frequency of deployments, changes increases
![Page 45: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/45.jpg)
Current Deployment Stats
• 4 Deployments running 24/7, 50 for development
• 20+ clusters per deployment
• 25+ software components deployed
• Hundreds of instances in production
• Less than 10 minutes to deploy from scratch
• Less than 4 minutes to restart hundreds of components
![Page 46: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/46.jpg)
dsh: Another AWS deployment tool
• Model-driven, describe desired state, run to make it so
• High performance due to parallelization
• Covers all layers of the stack – AWS, OS, Sumo Logic
• Easy to use and extend, scriptable CLI
• Developer-friendly, Scala-based, high-level APIs
![Page 47: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/47.jpg)
Example session
![Page 48: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/48.jpg)
Sie Ist Ein Model & Sie Sieht Gut Aus
• Model contains concepts– Deployment
– Cluster
– AWS Resources (Amazon S3, Amazon Elastic Load Balancing, Amazon
DynamoDB, Amazon RDS, etc.)
– Software assemblies
– AWS configuration (IAM users, security groups, etc.)
• Human-readable names: prod-index-5
![Page 49: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/49.jpg)
Model Snippet
![Page 50: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/50.jpg)
Model Snippet
![Page 51: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/51.jpg)
Differential Deployment
• Start by finding existing resources– Use tagging where it is available
– Name prefixes (“prod_xxx”) where it isn’t (security groups, IAM, …)
• Fix differences to model– Start “missing” instances
– Change security group rules, missing IAM users
• Proceed with caution– Never delete anything that holds data
– Amazon EBS, Amazon DynamoDB, Amazon S3, Amazon RDS
![Page 52: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/52.jpg)
Example Of Tag Usage
![Page 53: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/53.jpg)
Making It Fast
• Parallelize all the things– Upload to Amazon S3 while booting instances while creating IAM users
while setting up security groups while…
– Hyper-concurrent rolling restarts
![Page 54: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/54.jpg)
Hyper, Hyper
![Page 55: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/55.jpg)
Making It Fast
• Parallelize all the things– Upload to Amazon S3 while booting instances while creating IAM users
while setting up security groups while…
– Hyper-concurrent rolling restarts
• Fast enough for development– Write new code or fix a bug, compile locally
– Push code to development deployment and make it live
• Optimize data transfers– Use Amazon S3 hashes to only transfer new files
– Only upload changed JARs
![Page 56: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/56.jpg)
Making It Reliable
• Check prerequisites before you even try– Does Prod account have room for this many instances?
– Do I have the required permissions for the AWS APIs?
– Any model discrepancies I can’t automatically resolve? Too many Amazon EBS volumes?
• Handle common failures automatically– No m1.large in us-east-1b? Move Amazon EBS volumes to us-west-1c and
try there
– Hitting the AWS API rate limit? Throttle and try again
– SSH didn’t come up on the instance? Kill it and launch another
– Eventual consistency in AWS– query until it has the expected state (tags)
![Page 57: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/57.jpg)
Making It Secure
• Different AWS accounts
– Per developer
– Production
• account.xml
– All credentials for one AWS
account (AWS keys, SSH
keys)
– Password-protected
• IAM
– One user per Sumo
component
– Minimal IAM policy
– Inject AWS credentials
• Security Groups
– Part of the model
– Minimal privileges
![Page 58: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/58.jpg)
Making It Safe
• Let mistakes happen at most once
• Add safeguards to prevent operator mistakes
• Type in the deployment name before deleting anything
• Disallow risky operations in production (shutdown Prod)
• Don’t allow –SNAPSHOT code to be deployed in production
![Page 59: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/59.jpg)
Making It Easy
• Automate best practices– Distribute instances over availability zones evenly
– Register instances in Elastic Load Balancing and match AZs to
instances
– Tag all resources consistently
• Consistent naming– Generate SSH with logical names
![Page 60: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/60.jpg)
Making It Affordable
• Developers forget to shut stuff down– Deployment reaper automatically shuts down deployments
– Daily cost emails
• Per-team budgets– Manager responsible to
keep within budget
![Page 61: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/61.jpg)
Pitfalls
• Base AMI plus scripted installation prevents auto scaling
• Security group updates cause TCP disconnects
• This is fixed in the VPC stack, however
• Parallelism can cause stampedes (for example,
Amazon DynamoDB)
• Tagging API rate limits are easy to hit
![Page 62: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/62.jpg)
Loosely Coupled Components
![Page 63: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/63.jpg)
Loose Coupling In The Large
• A deployment is made up of many things
• Some of these things need to talk to each other
• Some of these things come and go
• Don’t pass in a huge list of static dependencies
• Start each application with one parameter
$ bin/receiver prod.service-registry.sumologic.com
![Page 64: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/64.jpg)
Service Registry
• Service Registry is a concept, enables discovery
• A client-side library accessing a Zookeeper cluster
• Services are abstracted into types
• Application provides and consumes different services
• Sumo Logic services (RPC)
• Third-party services (message bus)
• AWS services (Amazon ElastiCache, Amazon RDS)
![Page 65: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/65.jpg)
The Perils Of Horizontal Scale
• Scaling out a multi-tenant processing system
• 1000s of customers, 1000s of machines
• Parallelism is good, but locality has to be considered
• 1 customer distributed over 1000 machines is bad
• No single machine getting enough load for that customer
• Batches & shards will become too small
• Metadata and in-memory structures grow out of
proportion
![Page 66: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/66.jpg)
The Perils Of Horizontal Scale
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
![Page 67: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/67.jpg)
The Perils Of Horizontal Scale
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
![Page 68: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/68.jpg)
The Perils Of Horizontal Scale
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
1 2 1 2 1 2 1 2 1 2
1 2 1 2 1 2 1 2 1 2
1 2 1 2 1 2 1 2 1 2
1 2 1 2 1 2 1 2 1 2
1 2 1 2 1 2 1 2 1 2
![Page 69: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/69.jpg)
The Perils Of Horizontal Scale
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
1 2 3 4
5 6 7 8
![Page 70: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/70.jpg)
The Perils Of Horizontal Scale
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
1 1
1 1
![Page 71: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/71.jpg)
The Perils Of Horizontal Scale
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
1 1 2 2 2
1 1 2 2 2
![Page 72: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/72.jpg)
The Perils Of Horizontal Scale
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
1 3 4 1 3 4 2 3 5 2 3 5 2 3 6
7 7 5 8 5 8
1 3 4 1 3 4 2 3 5 2 3 5 2 3 6
7 7 5 8 5 8
7 7 5 8 5 8
5 8
5 8
5 8
6
6
6
![Page 73: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/73.jpg)
Customer Partitioning
• Each cluster elects a leader node via Zookeeper
• Leader runs the partitioning logic
•
• Partitioning written to Zookeeper
• Example: indexer node knows which customer’s message
blocks to pull from message bus
Set[Customer], Set[Instance] Map[Instance, Set[Customer]]
![Page 74: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/74.jpg)
Lessons Learned
![Page 75: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/75.jpg)
Some Tips On AWS S3
• Use the TransferManager class from the AWS Java SDK– Multi-part uploads and downloads
– Multi-threaded, overall latency reduction
• Use random prefixes for keynames in Amazon S3 buckets– Amazon S3 partitions by keyname prefix
• Endpoint URL for Amazon S3– s3.amazonaws.com might go to Virginia, or Pacific Northwest (!)
– If you are in us-east, use s3-external-1.amazonaws.com instead
http://aws.typepad.com/aws/2012/03/amazon-s3-performance-tips-tricks-seattle-hiring-event.html
![Page 76: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/76.jpg)
Elastic Block Store
• RAID-0 makes Amazon EBS faster– Use LVM RAID-0 if heavy I/O is required
– Align stripe sizes with file system block sizes
• Snapshotting Amazon EBS volumes– Snapshots eat performance
– Even for volumes with provisioned IOPS
• Overlapping snapshots– Can be scheduled too close together, like every minute
– I/Os start taking 30+ seconds
![Page 77: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/77.jpg)
Cost & Business Value
![Page 78: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/78.jpg)
Somebody Has To Pay For Lunch
• On-demand resources are very sexy
• Automation gives developers their own sandbox
• Compute is the most easily incurred cost
• You need an automated reaper
• Or just raise another round…
![Page 79: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/79.jpg)
Elasticity Is Not An Arbitrary Need
• At least in our system, there’s baseline load
• At least in our system, the cost is in compute
• Alert-based scaling can be safe & effective
• Measure your spend with tools that are out there
• We actually use Sumo Logic for that!
• Look for a moving average of resource consumption
• Buy Reserved Instances, don’t fret the instance types
![Page 80: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/80.jpg)
One More Thing
![Page 81: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/81.jpg)
Amazon CloudTrail
• Logs! From AWS! The eagle has landed!
• Amazon CloudTrail logs your API activity to Amazon S3
• Sumo Logic will read from Amazon S3, allow analysis
![Page 82: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/82.jpg)
![Page 83: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/83.jpg)
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
BDT401
![Page 84: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/84.jpg)
Chart Example
0% 20% 40% 60% 80% 100%
Category 1
Category 2
Category 3
Category 4
Axis Title
Series 1 Series 2 Series 3 Series 4
![Page 85: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/85.jpg)
Powerpoint Guidelines
Arial
Please do not use gradients, shadows or outlines on shape
elements in your presentation.
![Page 86: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/86.jpg)
PowerPoint Guidelines
When pasting content from another presentation please paste using “Destination Theme”
Windows Mac
Note: This works when copying entire slides from other presentations as long as the source presentation is also 16:9
![Page 87: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/87.jpg)
PowerPoint GuidelinesWhen pasting content Code into a Code template please use the “Keep Text Only Function” If any additional coloring needs to be done to your code type please do it after pasting it into your slide.
Windows Mac
![Page 88: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/88.jpg)
68k Assembly Code Sample; Syntax Test file for 68k Assembly code
; Some comments about this file
.D0 00000000
MS 2100 00000002
MM 2000;DI
LEA.L $002100,A1
MOVE.L #2,-(A1)
BSR $00002050
MM 2050;DI
MOVE.L (A1)+,D1
MOVE.L (A1),D2
ADD.L D1,D2
![Page 89: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/89.jpg)
Basic text content slide
• With Content– And more content
![Page 90: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/90.jpg)
![Page 91: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/91.jpg)
Title Slide #2
![Page 92: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/92.jpg)
Slide with two columns
![Page 93: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/93.jpg)
Slide with two columns and titles
![Page 94: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/94.jpg)
Slide with space for custom content
![Page 95: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/95.jpg)
![Page 96: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/96.jpg)
Side Content
Description or content with place for
image on the right
![Page 97: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/97.jpg)
Big picture slide
![Page 98: Using AWS To Build A Scalable Machine Data Analytics Service](https://reader031.vdocuments.mx/reader031/viewer/2022030402/587b448f1a28ab9c0e8b6675/html5/thumbnails/98.jpg)
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!