powerpoint presentation - amazon s3€¦ · ad tech live voting social media connected devices...
TRANSCRIPT
AWS Rapid Pace of Innovation
2009
Amazon RDS
Amazon VPC
Auto Scaling
Elastic Load
Balancing
+48
2010
Amazon SNS
AWS Identity
& Access
Management
Amazon Route 53
+61
2011
Amazon
ElastiCache
Amazon SES
AWS
CloudFormation
AWS Direct
Connect
AWS Elastic
Beanstalk
GovCloud
+82
Amazon
CloudTrail
Amazon
CloudHSM
Amazon
WorkSpaces
Amazon Kinesis
Amazon Elastic
Transcoder
Amazon
AppStream
AWS OpsWorks
+280
2013
Amazon SWF
Amazon Redshift
Amazon Glacier
Amazon
Dynamo DB
Amazon
CloudSearch
AWS Storage
Gateway
AWS Data
Pipeline
+159
2012
Since inception AWS has:
• Released 1111 new services and features
• Introduced more than 40 major new services
• Announced 45 price reductions
2008
+24Amazon EBS
Amazon
CloudFront
+454
2014
Amazon Cognito
Amazon Zocalo
Amazon Mobile
Analytics
*as of Nov 13, 2014
AWS Directory
Service
Amazon RDS for Aurora
AWS CodeDeploy
AWS Lambda
AWS Config
AWS Key Management
Service
AWS Service Catalog
Amazon EC2
Container Service
AWS CodePipeline
AWS CodeCommit
Everyday, AWS adds enough new server capacity to support
Amazon.com when it was a $7 billion global enterprise.
James Hamilton: Innovation at Scale Presentation re:Invent 2014
1.
2. 3. 4.
http://mvdirona.com/jrh/work/
https://www.youtube.com/
watch?v=JIQETrFC_SQ
We are driven to remove any all causes of failure.
Our goal is to make our operational performance indistinguishable from perfect.
“Based on our experience, I believe that we can be even more secure in the
AWS cloud than in our own data centers.” – Tom Soderstrom, CTO, NASA JPL
AWS provides the same, familiar approaches to security that companies have
been using for decades with increased visibility, control, and auditability.
Visibility
View your entire infrastructure with a click
Deep insight with
AWS CloudTrail
Control
You have sole
authority on where
data is stored
Shared
responsibility model
Auditability
3rd party validation
SOC 1 / SOC 2 / SOC 3
SSAE 16 / ISAE 3402
PCI DSS Level 1
DIACAP & FISMA
ISO 27001 / 9001 / 13485
ISO/TS 16949
FedRAMP (SM)
FISMA
HIPAA
ITAR
MPAA
CSA
FIPS 140-2
High volume / low margin businesses are in our core DNA
Trade CapEX for
variable expense
Our economies of
scale provide us
with lower costs
45 price
reductions
since 2006
Pricing model
choice to support
variable and
stable workloads
On-demand
Reserved
Spot
Save more money
as you grow bigger
Tiered pricing
Volume discounts
Custom pricing
5,000+ SIs & Consultants
3,000+ ISVs
22 Global Premier Tier partners
6 Enterprise-focused competencies
2,000+ products available for 1-click
deployment across 23 distinct product
categories
Customers run over 70M hours of
software per month
SDL Reduces Time to Market for New Customer Web Environments to Less
than an Hour by Using AWS
• Needed flexibility in provisioning and managing
customer web environments to support rapid growth
• Created a fully automated resource provisioning and
management platform running on AWS
• SDL reports accelerated time to market to less than
an hour
• By using AWS, SDL delivers a reliable, scalable, and
secure web offering to its customers
4Synergy is an AWS Consulting Partner at the Advanced Tier that helps
define cloud strategy and designs and implement cloud applications.
AWS provides us with flexibility and
scalability, both technically and
financially. It allows us to provide a
superior service without hefty
upfront infrastructure investments
Dennis van der Veeke
CTO, SDL
”
“
SDL Global Customer Experience Management provides
solutions for managing global customer experiences.
CustomerGauge Uses AWS to Reduce Time to Market by 50%
CustomerGauge automatically collects actionable
insights and improves customer experience based on
Net Promoter® metrics.
By using AWS, we scaled our
services to three regions from
100,000 to 3 million transactions
per month.
Alessio Nobile
Head of Technology
”
“ • CustomerGauge needed to scale its survey engine
and API by 300% to support its growing customer
base in Australia, Europe and the United States
• By using AWS, the company replicated its
environment in the United States and Australia in less
than 6 hours, meeting its scalability target
• CustomerGauge now report that it experiences near
99.9% availability and can update its service without
disrupting server instances and customer traffic.
Zeeman Scales to Support Traffic Spikes of 10x by Using AWS
Zeeman is a leading Dutch retail organization
with 1,200 physical shops and a turnover of
€ 500 million in the retail market.
Using AWS helps us prepare for a
highly variable workloads that
comes with spikes in visitor counts
around our marketing efforts.
• Needed a flexible solution to meet traffic spikes
immediately following commercial campaigns
• Built an e-commerce infrastructure on AWS that
scales with traffic fluctuations
• Designed to scale up by 10x seamlessly and
automatically following advertising campaigns
• By using AWS deployment and management
services, Zeeman continuously integrates updates
and patches several times a day
Jacques van der Bom
Manager, E-commerce
”
“
Unitt is an AWS Advanced Consulting Partner that develops,
builds, and manages cloud and hosting services.
FINRA handles approximately 30 billion market
events every day to build a holistic picture of
trading in the U.S.
Determisconduct by
enforcing the rules
Detectand prevent wrongdoing
in the U.S. markets
Disciplinethose who
break the rules
That Kind of Volume Comes with Challenges
Market volumes are volatile and steadily increasing
Exchanges are dynamically evolving
Regulatory rules are created and enhanced
New securities products are introduced
Market manipulators innovate
AWS Offered The Right Services For Our
Platform
Cloud PlatformAPIs at the right layer
Automated infrastructure deployment
Open source commitment
Operations Security
A Platform That Adapts to Market Dynamics
Data Integration
Hbase
Hadoop
MapReduce
Flexible Interactive
Queries
Hadoop
EMR
SQL/Hive
Fast Predefined Queries
Hbase/NoSQL
Hadoop
Predefined Datamarts
Surveillance Analytics
EMR
Hive
Web Applications
Analysts
Regulators
Data Management Services
Data Movement
Data Registration
Notification
Version Management
Job Management
Cluster Management
S3
Firms
Delivering Agility, Speed, and Cost Savings To
FINRA
Cost SavingsSpeedAgility
Efficient scale
Pay for what we use
Projected to save $10-$20M annually
Reduce query times
from hours to seconds
Respond quickly to market challenges
Glacier
S3 DynamoDB
RDS
EMR
Redshift
Data PipelineKinesis
Cassandra CloudSearch
Kinesis-
enabled
app
GlacierS3
DynamoDB
RDS
Kinesis
Spark
Streaming
EMRData Pipeline
Storm
Kafka
Redshift
Cassandra
CloudSearch
Kinesis
Connector
Kinesis
enabled app
Real-time processing of streaming data
High throughput
Elastic
Easy to use
Connectors for EMR, S3, Redshift, DynamoDB
Amazon
Kinesis
Amazon Web Services
AZ AZ AZ
Durable, highly consistent storage replicates dataacross three data centers (availability zones)
Aggregate andarchive to S3
Millions ofsources producing100s of terabytes
per hour
FrontEnd
AuthenticationAuthorization
Ordered streamof events supportsmultiple readers
Real-timedashboardsand alarms
Machine learningalgorithms or
sliding windowanalytics
Aggregate analysisin Hadoop or adata warehouse
Inexpensive: $0.028 per million puts
Amazon Kinesis Architecture
Aggregate All Data in S3 Surrounded by a collection of the right tools
EMR Kinesis
Redshift DynamoDB RDS
Data Pipeline
Spark StreamingCassandra Storm
Amazon
S3
Amazon S3
App/Web Tier
Client Tier
Data TierDatabase & Storage Tier
Search
Hadoop/HDFS
Cache
Blob Store
SQL NoSQL
Database & Storage Tier
Amazon RDSAmazon
DynamoDB
Amazon ElastiCache
Amazon S3
Amazon
Glacier
Amazon CloudSearch
Amazon EMR
Fully-managed NoSQL database service
Built on solid-state drives (SSDs)
Consistent low latency performance
Any throughput rate
No storage limits
Amazon
DynamoDB
WRITES
continuously replicated andpersisted (SSD)
READS
strongly or eventually consistent (option)
better than
99.999%worldwide uptime
Provably Highly
Durable Available
File & PhotoSharing
OnlineGaming
Ad Tech Live Voting
SocialMedia
ConnectedDevices
State Management
Video Streaming
MobileMessaging
Backup & Restore
Publishing Mapping
(Some) DynamoDB Use Cases
Relational Databases
Fully managed = low admin
Trillions of I/O requests/month
Aurora, MySQL, Oracle, SQL Server,
Postgres
Amazon
RDS
• Manageability Rapid deployment with pre-configured parameters Patch Management Monitoring and Metrics
• Availability and Data Durability Automated Backups and Point-In-Time-Recovery DB Snapshots Automatic Host Replacement (Single-AZ) Multi-AZ deployments
• Scalability Push-Button Scaling
• Storage, Memory and Compute Read Replicas
Key Features
Columnar data warehouse
ANSI SQL compatible
Massively parallel
Petabyte scale
Fully-managed
Very cost-effective
Amazon
Redshift
Hadoop/HDFS clusters
Hive, Pig, Impala, HBase
Easy to use; fully managed
On-demand and spot pricing
Tight integration with S3,
DynamoDB, and Kinesis
Amazon
Elastic
MapReduce
A look at how it works
Months of user history Common misspellings
Data Analyzed Using EMR:
Weste
nWistin
Westa
nWhestin
Automatic spelling corrections
Months of user search data
Search terms
Misspellings
Final click throughs
Yelp web site log data goes into Amazon S3
Amazon S3
Hadoop Cluster
Amazon EMRAmazon S3
All 200 nodes of the cluster simultaneously look for common misspellings
Westen
Wistin
Westan
Hadoop Cluster
Amazon EMRAmazon S3
A map of common misspellings and suggested corrections are loaded back into Amazon S3.
Westen
Wistin
Westan
Then the cluster is shut down Yelp only pays for the time they used it
Hadoop Cluster
Amazon EMRAmazon S3
Each of Yelp’s 80 Engineers Can Do This Whenever They Have a Big Data Problem
spins up over
250 Hadoop
clusters per week in EMR.
Amazon EMRAmazon S3
Data Innovation Meets Action at Scale
at NASDAQ OMX
• NASDAQ’s technology powers more than 70 marketplaces in 50 countries
• NASDAQ’s global platform can handle more than 1 million messages/second at
a median speed of sub-55 microseconds
• NASDAQ own & operate 26 markets including 3 clearinghouse & 5 central securities repositories
• More than 5,500 structured products are tied to NASDAQ’s global indexes with the notional value of at least $1 trillion
• NASDAQ powers 1 in 10 of the world’s securities transactions
NASDAQ’s Big Data Challenge
• Archiving Market Data
– A classic “Big Data” problem
• Power Surveillance and Business Intelligence/Analytics
• Minimize Cost
– Not only infrastructure, but development/IT labor costs too
• Empower the business for self-service
NASDAQ’s Legacy Solution
• On-premises MPP DB
– Relatively expensive, finite storage
– Required periodic additional expenses to add more storage
– Ongoing IT (administrative) human costs
• Legacy BI tool
– Requires developer involvement for new data sources, reports,
dashboards, etc.
New Solution: Amazon Redshift
• Cost Effective
– Redshift is 43% of the cost of legacy
• Assuming equal storage capacities
– Doesn’t include IT ongoing costs!
• Performance
– Outperforms NASDAQ’s legacy BI/DB solution
– Insert 550K rows/second on a 2 node 8XL cluster
• Elastic
– NASDAQ can add additional capacity on demand, easy to grow their cluster
• Amazon Redshift partner
– http://aws.amazon.com/redshift/partn
ers/pentaho/
• Self Service
– Tools empower BI users to integrate
new data sources, create their own
analytics, dashboards, and reports
without requiring development
involvement
• Cost effective
New Solution: Pentaho BI/ETL
Net Result
• New solution is cheaper, faster, and offers capabilities that NASDAQ
didn’t have before
– Empowers NASDAQ’s business users to explore data like they never
could before
– Reduces IT and development as bottlenecks
– Margin improvement (expense reduction and supports business
decisions to grow revenue)
The Cloud Enables Specialization
workload = ( access patterns )
fit the infrastructure
to the workload
Spark
Streaming,
Storm
Amazon
Redshift Spark,
Impala,
Presto
Hive
Amazon
Redshift
Hive
Spark,
Presto
Amazon
Kinesis/
Kafka
Amazon
DynamoDBAmazon S3Data
Hot ColdData TemperatureQ
ue
ry L
ate
nc
y
Low
HighAnswers
HDFS
Hive
Native
Client
Spark
Streaming
Hive
Amazon Kinesis / KafkaData
Answers
Apache Storm Native Client
Amazon
DynamoDB
Native
Client
http://aws.amazon.com/marketplace
Big Data Case Studies
Learn from other AWS customers
aws.amazon.com/solutions/case-studies/big-data