Serverless Architectures with AWS Lambda and MongoDB Atlas
Sig NarváezSr. Solutions [email protected] @SigNarvaez
Serverless?• Landscape• Use cases
Going Serverless• What changes?• Considerations• MongoDB Atlas
AWS & MongoDB Atlas• Simple API for Customer Single View• Lambda & API Gateway• MongoDB Atlas & Compass• Postman
Agenda
Serverless
Big Iron Commodity Hardware
Virtualized
Containers
Functions
Where will my code run?
Serverless Frameworks and Platforms
https://github.com/serverless/serverless
https://www.zappa.io/
Chalice (awslabs)https://github.com/awslabs/chalice
Frameworks for Cloudproviders
On-Prem PaaS – now offering FaaS
1. Cloud services have matured
2. BaaS “SaaS-ification”
3. API’s are the glue
4. Containers – now per function
5. SysOps DevOps NoOps Less Ops, More Engineering
5 factors fuelling Serverless Computing
https://www.forbes.com/sites/janakirammsv/2016/02/28/five-factors-that-are-fueling-serverless-computing-part-1
Thoughtworks Technology Radar
Scheduled Jobs• Sequencing / Orchestration (AWS Steps?)
Data Quality• Trigger Identify Pass to function
Micro or Nano services• Clicks or Taps
Event and IoT processing• Don’t worry about scaling App Servers
Lightweight API’s• Focus of today!
Good fit for Serverless?
Serverless
Microservices
Before and after
Container-based Microservice
Payments Service
Product Catalog Service Shopping
CartService
Domainshttps://www.mongodb.com/blog/post/serverless-architectures-the-evolution-of-cloud-computing
Serverless Microservice
CommandQueryResponsibilitySegregation
Think about:Fine or course grainedShared logicStart-up time!
PackagingDeploymentVersioning
CQRS pattern on Serverless Microservices
GETAPI
PUT PATCH POST DELETE …
API
API Key
API Key
Lambda Function(s)
Lambda Function(s)
Code
Code
Lambda Function(s) VPC
Peering
Shape• Person• Insurance Policies
• Shape changes per policy type• Addresses
Operations via API• GET Customers with soon-to-expire
policies, within a geo radius• GET Customers / by SSN, id, etc.• PATCH Update basic contact info
(cell, email, …)
Customer Single View - Insurance Industry (hypothetical)
High-level architecture of a single view platform
MongoDB Atlas &
AWSBuild it!
Required MongoDB Services – Atlas!
mgeneratejs• https://
github.com/rueckstiess/mgeneratejs• npm install -g mgeneratejs• Create template – generate data• Upload to Atlas via mongoimport
• Hint: get connection string from Atlas UI!• Browse with Compass
Generate dataset
Template (InsuranceC360_Customers.json)
mgeneratejs -n 100 InsuranceC360_Customers.json | mongoimport --host ”YOUR ATLAS CLUSTER" --numInsertionWorkers 4--db WebinarCustomerSingleView --collection Customers --authenticationDatabase admin --ssl --username YOURUSER --password YOURPASSWORD
IAM• Role with Lambda execute
policies
VPC• VPC• Security Groups – traffic rules• Internet Gateway – outside
communication• VPC Peering Connection -
Route Table
Required AWS Services
Lambda• VPC, Security Group and IAM
role• Develop inline or upload
deployment package (.zip)• Use MongoDB Driver –
connect with MongoDB Atlas
API Gateway• API definition• API Keys & Usage Plans• Resources and HTTP Methods• Map Routes to Lambda
functions
VPC
MongoDB Atlas• Provision a Cluster – M10+ need an assigned AWS region for VPC peer• Same AWS region – (I will use us-west-2)• Initiate VPC peer with AWS
AWS VPC• Accept incoming Peering Connection• Update Route Table
EC2• Install MongoDB• Test connection from the MongoDB Shell to ensure VPC Peer is working• Optional but highly recommended – ensure VPC Peering is working before
proceeding to Lambda
MongoDB Atlas peered with your AWS VPC
VPC Peering
Atlas AWS
Verify VPC Peer works
Security Group
Peering Connections
Lambda
Role with lambda permissions (IAM)
Code packagingfrom __future__ import print_function
import jsonimport pymongo
print('Loading function')print(’=== CONNECTING TO MONGODB ATLAS ===')connstr = ”ENTER YOUR MONGODB ATLAS CONNECTION HERE"MONGOCLIENT = pymongo.MongoClient(connstr, readPreference=’secondaryPreferred’)
def GET_lambda_handler(event, context):
… implement GET logic
def POST_lambda_handler(event, context):
… implement POST logic http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
Lambda functions
Upload & configure function
The handler function
The role with lambda permissions
The VPC (peered with Atlas)
The security group that allows traffic
At least 2 subnets
API Gateway
Read API – GET /api/v1/customers
CUD API - PATCH /api/v1/customers
Deploying the API
Access and throttling via API Keys
Test!
Test with Postman
Load test too!
AWS CloudWatch
Connections and containers …..
http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html
… AWS Lambda maintains the container for some time in anticipation of another Lambda function invocation. … the service freezes the container after a function completes, and thaws the container for reuse. If AWS Lambda chooses to reuse the container, this has the following implications:
- Any declarations in your Lambda function code (outside the handler code, see Programming Model) remains initialized, providing additional optimization when the function is invoked again. For example, if your Lambda function establishes a database connection, instead of reestablishing the connection, the original connection is used in subsequent invocations. You can add logic in your code to check if a connection already exists before creating one.
MongoDB Atlas Monitoring and Alerts
MongoDB Compass
Done!…
But what about?
Scaling?Scaling Lambda
No user intervention required - Default safety throttle of 100 concurrent executions per account per region.
Functions invoked synchronously throw 429 error code. Functions invoked asynchronously can absorb reasonable bursts for approx. 15-30 minutes. If exhausted, consider using Simple Queue Service (SQS) or Simple Notification Service (SNS) as the Dead Letter Queue (DLQ).
Read more at https://aws.amazon.com/lambda/faqs/
Scaling MongoDB Atlas
On-DemandZero downtimeUpscale/Downscale:• Instance size• Storage size• IOPS• Replication
factor.
Pricing?Lambda Costs
Cost depends on requests (per million), request time, memory (GB) allocated to each function.
First 1 million requests per month free - $0.20 per 1 million requests thereafter. $0.00001667 for every GB-second used.
Additional AWS services imply cost (e.g. API Gateway, …)
Read more at https://aws.amazon.com/lambda/pricing/
MongoDB Atlas Costs
Cost depends on instance size, storage, iops, replication factor and backup retention.
M0 free – great for you (no VPC peering, use IP whitelist)M10 starts at $0.08/hr – great for team DevM30 starts at $0.54.hr – great for Production
Read more at https://www.mongodb.com/cloud/atlas/pricing
Connections to MongoDB Atlas
Encrypt using AWS KMS – see this blog post: https://www.mongodb.com/blog/post/serverless-development-with-nodejs-aws-lambda-mongodb-atlas
Container freeze & recycle?• Connection outside lambda function
helps• On scale new containers, new
connections• Ok if API is used in bursts, but
maybe not ok if used seldomly
• If not?
Others?Local development? Lambda emulators
• python-lambda-local at https://pypi.python.org/pypi/python-lambda-local
• lambda-local (node.js) at https://www.npmjs.com/package/lambda-local
Serverless frameworks – evaluate them! … F500’s are!• Serverless Framework• Zappa• Chalice• More! - https://thenewstack.io
/tns-guide-serverless-technologies-best-frameworks-platforms-tools/
Façade Serverless Functions – logic querying backend API
Backend Traditional stateful layer - CRUD API to Data Stores
Would this be a Serverless
Architecture ??
Customer Single View - Insurance Industry (hypothetical)
High-level architecture of a single view platform
Stateful API Service Layer
Serverless Architectures with AWS Lambda and MongoDB Atlas
Q&AUse code "Sig" for 25% off!Parties of 3+ get addtl 25%
Sig NarváezSr. Solutions [email protected] @SigNarvaez
