amazon documentdb (with mongodb compatibility) deep dive 1 - am… · aurora communitycommercial...
TRANSCRIPT
© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.
Amazon DocumentDB (with MongoDB compatibility) Deep Dive
Jeff DuffyDocumentDB Specialist SA
© 2020, Amazon Web Services, Inc. or its Affiliates.
AWS: Purpose-built databases
Relational Key-value Document In-memory Graph
Amazon DynamoDB
Amazon Neptune
Amazon Relational Database Service
(RDS)
Aurora CommercialCommunity
Amazon ElastiCache
Amazon Keyspaces
Amazon DocumentDB
Time-series Ledger
Amazon Timestream
Amazon Quantum
Ledger DatabaseRedis Memcached
Wide Column
© 2020, Amazon Web Services, Inc. or its Affiliates.
Amazon DocumentDBFast, scalable, and fully managed MongoDB-compatible database service
Fast Scalable Fully managed MongoDB compatible
Millions of requests per second with millisecond latency
Separation of compute and storage scales both
independently; scale out to 15 read replicas in minutes
Managed by AWS: no hardware provisioning; auto patching, quick setup,
secure, and automatic backups
Compatible with MongoDB 3.6; use the same SDKs, tools, and
applications with Amazon DocumentDB
Purpose-built document database engineered for the cloud
© 2020, Amazon Web Services, Inc. or its Affiliates.
Customers
Learn more: https://aws.amazon.com/documentdb/customers/
© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.
Document Databases
© 2020, Amazon Web Services, Inc. or its Affiliates.
(Client)
(App) (Database)
JSON
!=Evolution of document databases
Relational
JSON became the de facto
data interchange format
Friction when converting JSON to the relational
model
Object-relational mappers (ORMs) were created to
minimize friction
Document databases solved
the problem
JSON
© 2020, Amazon Web Services, Inc. or its Affiliates.
Documents (objects/JSON) are common application data models
Documents map naturally to how humans model data
Document databases store JSON-like documents
Document databases provide flexible schema and indexing
Ad hoc querying and aggregations
Denormalized data
modelNormalized data model
Why document databases?
© 2020, Amazon Web Services, Inc. or its Affiliates.
Use cases for document data
Mobile
Retail and marketing
User profilesCatalog
Content management
Personalization
© 2020, Amazon Web Services, Inc. or its Affiliates.
{userid: 181276,username: "sue1942",name: {first: "Susan",
last: "Benoit"}}
{userid: 181276,username: "sue1942",name: {first: "Susan",
last: "Benoit"},ExplodingSnails: {
hi_score: 3185400,global_rank: 5139,bonus_levels: true
},promotions: ["new user","5%","snail lover"]
}
{userid: 181276,username: "sue1942",name: {first: "Susan",
last: "Benoit"},ExplodingSnails: {
hi_score: 3185400,global_rank: 5139,bonus_levels: true
}}
Document Use Case: Gaming User Profiles
© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.
Challenges scaling document databases
© 2020, Amazon Web Services, Inc. or its Affiliates.
API
Query processor
Caching
Logging
Storage
Application
Not designed for the cloud
Single monolithic architectures
Scale monolithicallyFail monolithically
Traditional Database Architecture Challenges
© 2020, Amazon Web Services, Inc. or its Affiliates.
Replication
Challenge: Add read capacity on-demand
Node 1 Node 2 Node 3 Node 4Disk Disk Disk Disk
© 2020, Amazon Web Services, Inc. or its Affiliates.
Replication
Challenge: Recover quickly from node failure
Node 1 Node 2 Node 3 Node 3’Disk Disk Disk Disk
© 2020, Amazon Web Services, Inc. or its Affiliates.
Challenge: Scale storage as data grows
Node
Storage Volume
Node
Storage Volume
Storage Volume
Node
© 2020, Amazon Web Services, Inc. or its Affiliates.
Operational Node
Backup Node
Challenge: Backup data without affecting performance
Snapshot Snapshot
© 2020, Amazon Web Services, Inc. or its Affiliates.
Challenge: Data Durability
Node
Storage Volume
Node
Storage Volume Storage Volume
NodeReplication Replication
© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.
Amazon DocumentDBPurpose-built and engineered
for the cloud
© 2020, Amazon Web Services, Inc. or its Affiliates.
API
Query processor
Caching
Logging
Storage
Scale compute
Scale storage
Decouple compute and storage
Compute layer
Storage layer
© 2020, Amazon Web Services, Inc. or its Affiliates.
Instance (Replica)
Distributed storage volume
AZ1 AZ2 AZ3
Read
s
Instance (Primary)
Instance (Replica)
Read
s Writes Re
ads
Compute
Storage
Amazon DocumentDB: Cloud Native Architecture
© 2020, Amazon Web Services, Inc. or its Affiliates.
Instance (Replica)
Distributed storage volume
AZ1 AZ2 AZ3
Read
s
Instance (Primary)
Instance (Replica)
Read
s Writes Re
ads
Instance (Replica)
Read
s
Compute
Storage
Challenge: Add read capacity on-demand
~8-10 mins
© 2020, Amazon Web Services, Inc. or its Affiliates.
Distributed storage volume
Challenge: Quickly recover from node failure
~8-10 mins
~30 secs
Read
s
Writes
Read
s
Read
s WritesRe
ads
Read
s
Instance (Replica)
Instance (Primary)
Instance (Replica)Instance (Primary)
Instance (Replica)
© 2020, Amazon Web Services, Inc. or its Affiliates.
Distributed storage volume
Challenge: Scale storage as data grows
Grows Automatically from
10GB-64TB
Compute
Storage
Distributed storage volume
Storage
Read
s
WritesRe
ads
Read
s
Instance (Replica)
Instance (Primary)
Instance (Replica)
© 2020, Amazon Web Services, Inc. or its Affiliates.
Distributed storage volume
Instance (Replica)
Read
s
Instance (Primary)
Instance (Replica)
Read
s
Writes
Amazon S3
Continuous Backups(PITR)
Snapshots (Automated and Manual)
Read
s
Compute
Storage
Challenge: Backup data without affecting performance
© 2020, Amazon Web Services, Inc. or its Affiliates.
Distributed storage volume
AZ1 AZ2 AZ3
Instance (Replica)
Read
s
Instance (Primary)
Instance (Replica)
Read
s
Writes
Read
s
Instances:
Environment:
Durability: highly durable
1
dev/testproduction
2
99%Availability goal: 99.9%
3
99.99%
Challenge: Data Durability
© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.
Service overview
© 2020, Amazon Web Services, Inc. or its Affiliates.
Automated maintenance
Automated upgrades and patching
Failed instances are automatically replaced
Automatic failover and recovery
Fully managed
AWS Integration
CloudWatch, CloudTrail, CloudFormation, VPC,
Secrets Manager
Pay-as-you-go pricing
Per-second instance billing, no long-term
commitment
“Our engineering teams now spend less time on operations like backup scripts, scale testing, and managing high availability and instead are able to focus on developing new capabilities for our customers.”
© 2020, Amazon Web Services, Inc. or its Affiliates.
MongoDB compatible
MongoDB 3.6 Same drivers, tools
Replica sets
Compatible with MongoDB Community Edition 3.6
Use the same MongoDB drivers and tools with Amazon DocumentDB
Read scaling is easy with replica set emulation
“Getting started with DocumentDB was also simple and we migrated our application in a couple of days without needing to make any meaningful code changes. Everything just worked.”
© 2020, Amazon Web Services, Inc. or its Affiliates.
Why choose the MongoDB API?
Source: http://db-engines.com
• Over 60 million downloads
• Easy for developers to get started
• MongoDB API is powerful
• Managing MongoDB is hard
© 2020, Amazon Web Services, Inc. or its Affiliates.
Scale out in minutes
Scale up in minutes
Autoscalestorage
Load balancing
Scale to 15 read replicas
Scale from 16 to 768 GiB
of RAM
Storage automatically grows from
10 GB to 64 TB
Scale reads across replicas
Scalable
© 2020, Amazon Web Services, Inc. or its Affiliates.
Launch
JAN
2019
MAR
201
9
Frankfurt
JUN
201
9
MAY
201
9
Secrets Manager
JUL
2019
Tokyo
Per-second Billing
AUG
2019
Sydney
FEB
2019
DDLAuditing
APR
2019
… with more to come in 2020!
OCT
201
9
Start/Stop Cluster
DEC
2019
FEB
2020
London
Canada
APR
2020
Slow query logger
Glue ETL
MAR
202
0
RBACAggregation
Operators
Change Streams
Mumbai
ParisSingapore
Aggregation Operators
Deletion ProtectionSeoul
Aggregation Operators
Working Backwards
© 2020, Amazon Web Services, Inc. or its Affiliates.
Scale Reads
Sharding motivator
Scale Writes
Scale Storage
Add up to 15 read replicas
Scale vertically with very low impact
DocumentDB approach
Storage scales automatically to 64TB
Why not shard?
© 2020, Amazon Web Services, Inc. or its Affiliates.
Security and compliance
VPC Only Encrypted by default
Safe defaults Compliance
Strict network isolation with Amazon Virtual Private Cloud
(VPC)
Encryption at rest with AWS KMS
Encryption in transit with TLS
Best practices are the defaults
PCI DSS,ISO 9001, 27001, 27017,
and 27018,SOC 1, 2 & 3, HIPAA
© 2020, Amazon Web Services, Inc. or its Affiliates.
Backup and recovery
Automatic backups 35 days of PITR
Archive snapshots
Automatic, incremental, and continuous backups
Backups do not affect database performance
Point-in-time recovery (PITR) for up to 35 days
Keep snapshots for as long as you need
No performance impact
"Adopting Amazon DocumentDB is a game-changer because we offload management, security, and backup of our MongoDB databases to AWS. With Amazon DocumentDB, we can add or scale instances in minutes, regardless of data size. Further, we get automatic backups and point-in-time restore capabilities, which far exceed other managed DB services at less cost.”
© 2020, Amazon Web Services, Inc. or its Affiliates.
Distributed storage volume
Read
s
Read
s Writes
Amazon S3
Read
s
Compute
Storage
Instances: Size/hr * count (db.r5.large $0.277/hr)
I/O: Count ($0.20/million)
Storage: GB/mo
Backup: GB/mo(100% Free! then $0.021/GB)
Pricing (us-east-1)
($0.10/GB)
© 2020, Amazon Web Services, Inc. or its Affiliates.
Offline migration using MongoDB utilities:Get started quickly, great for POCs
Online migration using AWS DMS Near-zero downtime migration
Hybrid migration leverages both solutionsBest of both worlds
Migration GuideLearn how to migrate from MongoDB to Amazon DocumentDB
All options support MongoDB on-premises and EC2, for both replica sets and sharded clusters
https://docs.aws.amazon.com/documentdb/latest/developerguide/docdb-migration.html
© 2020, Amazon Web Services, Inc. or its Affiliates.
Purpose builtThe right tool for the right
job
Thank you