Webinar: Serverless Architectures with AWS Lambda and MongoDB Atlas

Download Webinar: Serverless Architectures with AWS Lambda and MongoDB Atlas

Post on 06-Apr-2017




9 download

Embed Size (px)


<p>PowerPoint Presentation</p> <p>Serverless Architectures with AWS Lambda and MongoDB Atlas</p> <p>Sig NarvezSr. Solutions Architectsig@mongodb.com @SigNarvaez</p> <p>Hello everyone, thanks for attending today, and thanks Peter for the introduction. </p> <p>I really hope that you enjoy this webinar, and if you have questions, we will have a Q&amp;A session at the end, and during the webinar, please feel free to enter questions in the chat box. Also here is my contact information, or you can follow me on twitter.</p> <p>Serverless?LandscapeUse cases</p> <p>Going ServerlessWhat changes?ConsiderationsMongoDB Atlas</p> <p>AWS &amp; MongoDB AtlasSimple API for Customer Single ViewLambda &amp; API GatewayMongoDB Atlas &amp; CompassPostman</p> <p>Agenda</p> <p>Ok, lets get started. Today well learn about serverless computing, and understand how did we get here, where we think it is applicable, and why MongoDB Atlas, our DBaaS is also part of this trend. </p> <p>The first 5 mins or so will be an introduction, followed by hands-on demo of building a simple (hypothetical) API powered by services of the AWS cloud and MongoDB Atlas.</p> <p>Serverless</p> <p>Big IronCommodity HardwareVirtualizedContainersFunctionsWhere will my code run? </p> <p>As developers we think about our code, and where will it run, and as we have seen throughout time, the host of our code is getting smaller and smaller every time, and also easier and easier to manage.</p> <p>Recently, we got on board the container ship. Where we realized, hey, I dont need a full VM to run my code! - and now, we are taking that a step further by thinking about functions, and let something else run the container.</p> <p>And this is where the 3 major cloud vendors are now focusing on, FaaS: AWS Lambda, Google Cloud Functions, Azure Functions AWS Lambda being the leader, Google Cloud Functions is on Beta / Early Adopter and it was a major focus of the Google Next17 conference just 2 weeks ago.</p> <p>Serverless Frameworks and Platforms</p> <p>https://github.com/serverless/serverless</p> <p>https://www.zappa.io/Chalice (awslabs)https://github.com/awslabs/chalice</p> <p>Frameworks for CloudprovidersOn-Prem PaaS now offering FaaS</p> <p>Frameworks are proliferating, with Serverless being the leader and based on AWS Lambda, although I believe they will offer cross-cloud deployments. Zappa is python based and works with Django, Flask and otthersChalice is a Python Microframework from AWS directly.</p> <p>But FaaS is not only public cloud, the On-Prem PaaS providers like IBM BlueMix, RedHat OpenShift and Azure Stack are also offering FaaS</p> <p>Cloud services have matured</p> <p>BaaS SaaS-ification </p> <p>APIs are the glue</p> <p>Containers now per function</p> <p>SysOps DevOps NoOps Less Ops, More Engineering</p> <p>5 factors fuelling Serverless Computinghttps://www.forbes.com/sites/janakirammsv/2016/02/28/five-factors-that-are-fueling-serverless-computing-part-1</p> <p>The first three everyone can agree on</p> <p>Cloud services are now mature and probably offer all the services that developers needWe live in a state of SaaS-ification and Serverless Computing is the next stepSince everything is a service, APIs have become the glue, and they are inseparable from Serverless computing, as functions are headless, and APIs become their Faade.We are still on the Containerization journey, and as a matter of fact, your serverless functions are hosted on containers (1-1), but now managed by the cloud provider - yet just another operational burden turned into a managed service, and being monetizedBut the most important reason of why Serverless computing has now caught the attention of CTOs and CIOs is the promise of yet again reducing operational cost, and increasing developer productivity. Less Ops, More Engineering</p> <p>Thoughtworks Technology Radar</p> <p>Last year, serverless architecures moved from Assess to Trial category</p> <p>Try this on a project that can handle the risk dont forget to understand the capabilities and where it is a good fit</p> <p>Scheduled JobsSequencing / Orchestration (AWS Steps?)Data QualityTrigger Identify Pass to functionMicro or Nano servicesClicks or TapsEvent and IoT processingDont worry about scaling App ServersLightweight APIsFocus of today!</p> <p>Good fit for Serverless?</p> <p>Serverless</p> <p>How do I think of Serverless? - As the snap chat of computing ?</p> <p>MicroservicesBefore and after</p> <p>Container-based Microservice </p> <p>Payments ServiceProduct Catalog ServiceShopping CartServiceDomains</p> <p>https://www.mongodb.com/blog/post/serverless-architectures-the-evolution-of-cloud-computing</p> <p>Serverless Microservice </p> <p>CommandQueryResponsibilitySegregationThink about:Fine or course grainedShared logicStart-up time!</p> <p>PackagingDeploymentVersioning</p> <p>With Serverless things get more granular</p> <p>I used to have to manage deployments and scale C &amp; Q separately staff to do so and so on no more. With AWS API Gateway I can manage API keys, security, throttling and so on, and now with serverless functions, no need to worry about scale and availability and routing and DNS mappings and so on, I just focus on my application code and logic.With MongoDB Atlas likewise, I dont need to worry about managing servers, patching and so on. I focus on how the database benefits my application and my consumers, and I can upscale or downscale as necessary</p> <p>Startup Latency: 10ms to 2 mins!! - some languages add more time: Javascript/Python 10-100ms, JVM, &gt; 10 secsMore noticeable if infrequent callsor X calls per Function instance and you have an influx- Hack of pinging - reqires a server!HOW TO SCALE?</p> <p>CQRS pattern on Serverless Microservices GETAPIPUT PATCH POST DELETE APIAPI KeyAPI Key</p> <p>Lambda Function(s)Lambda Function(s)CodeCodeLambda Function(s)</p> <p>VPC Peering</p> <p>Own deploymentOwn API KeysResources &amp; Methods may map to the same Lambda FunctionLambda Function may map to your same code asset</p> <p>ShapePersonInsurance PoliciesShape changes per policy typeAddresses</p> <p>Operations via APIGET Customers with soon-to-expire policies, within a geo radiusGET Customers / by SSN, id, etc.PATCH Update basic contact info (cell, email, )Customer Single View - Insurance Industry (hypothetical) </p> <p>High-level architecture of a single view platform</p> <p>Note: Mention that Atlas, Lambda &amp; VPC have to be in the same region</p> <p>MongoDB Atlas &amp; AWSBuild it!</p> <p>Required MongoDB Services Atlas!</p> <p>mgeneratejshttps://github.com/rueckstiess/mgeneratejsnpm install -g mgeneratejsCreate template generate dataUpload to Atlas via mongoimportHint: get connection string from Atlas UI!Browse with Compass</p> <p>Generate dataset</p> <p>Template (InsuranceC360_Customers.json)mgeneratejs -n 100 InsuranceC360_Customers.json | mongoimport --host YOUR ATLAS CLUSTER" --numInsertionWorkers 4--db WebinarCustomerSingleView --collection Customers --authenticationDatabase admin --ssl --username YOURUSER --password YOURPASSWORD</p> <p>Note: Mention that Atlas, Lambda &amp; VPC have to be in the same region</p> <p>IAMRole with Lambda execute policies</p> <p>VPCVPCSecurity Groups traffic rulesInternet Gateway outside communicationVPC Peering Connection - Route Table</p> <p>Required AWS ServicesLambdaVPC, Security Group and IAM roleDevelop inline or upload deployment package (.zip)Use MongoDB Driver connect with MongoDB Atlas</p> <p>API GatewayAPI definitionAPI Keys &amp; Usage PlansResources and HTTP MethodsMap Routes to Lambda functions</p> <p>VPC</p> <p>MongoDB AtlasProvision a Cluster M10+ need an assigned AWS region for VPC peerSame AWS region (I will use us-west-2)Initiate VPC peer with AWS</p> <p>AWS VPCAccept incoming Peering ConnectionUpdate Route Table</p> <p>EC2Install MongoDBTest connection from the MongoDB Shell to ensure VPC Peer is workingOptional but highly recommended ensure VPC Peering is working before proceeding to Lambda</p> <p>MongoDB Atlas peered with your AWS VPC</p> <p>Note: Mention that Atlas, Lambda &amp; VPC have to be in the same region</p> <p>VPC Peering</p> <p>AtlasAWS</p> <p>Verify VPC Peer works</p> <p>Security Group</p> <p>Peering Connections</p> <p>Lambda</p> <p>Role with lambda permissions (IAM)</p> <p>Code packagingfrom__future__importprint_function</p> <p>importjsonimportpymongo</p> <p>print('Loadingfunction')print(=== CONNECTING TO MONGODB ATLAS ===')connstr=ENTER YOUR MONGODB ATLAS CONNECTION HERE"MONGOCLIENT=pymongo.MongoClient(connstr, readPreference=secondaryPreferred)</p> <p>defGET_lambda_handler(event,context):</p> <p> implement GET logic</p> <p>defPOST_lambda_handler(event,context):</p> <p> implement POST logic</p> <p>http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html</p> <p>Lambda functions</p> <p>Upload &amp; configure functionThe handler functionThe role with lambda permissionsThe VPC (peered with Atlas)The security group that allows trafficAt least 2 subnets</p> <p>API Gateway</p> <p>Read API GET /api/v1/customers</p> <p>CUD API - PATCH /api/v1/customers</p> <p>Deploying the API</p> <p>Access and throttling via API Keys</p> <p>Test!</p> <p>Test with Postman</p> <p>Load test too!</p> <p>Mention why US-EAST-1</p> <p>AWS CloudWatch</p> <p>Connections and containers .. http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html AWS Lambda maintains the container for some time in anticipation of another Lambda function invocation. the service freezes the container after a function completes, and thaws the container for reuse. If AWS Lambda chooses to reuse the container, this has the following implications:</p> <p>- Any declarations in your Lambda function code (outside the handler code, see Programming Model) remains initialized, providing additional optimization when the function is invoked again. For example, if your Lambda function establishes a database connection, instead of reestablishing the connection, the original connection is used in subsequent invocations. You can add logic in your code to check if a connection already exists before creating one.</p> <p>http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html</p> <p>Recurring ping hack serverless?</p> <p>MongoDB Atlas Monitoring and Alerts</p> <p>MongoDB Compass</p> <p>Done!But what about?</p> <p>Scaling?Scaling Lambda</p> <p>No user intervention required - Default safety throttle of 100 concurrent executions per account per region. </p> <p>Functions invoked synchronously throw 429 error code. Functions invoked asynchronously can absorb reasonable bursts for approx. 15-30 minutes. If exhausted, consider using Simple Queue Service (SQS) or Simple Notification Service (SNS) as the Dead Letter Queue (DLQ).</p> <p>Read more at https://aws.amazon.com/lambda/faqs/</p> <p>Scaling MongoDB Atlas</p> <p>On-DemandZero downtimeUpscale/Downscale:Instance sizeStorage sizeIOPSReplication factor.</p> <p>http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html</p> <p>Recurring ping hack serverless?</p> <p>Pricing?Lambda Costs</p> <p>Cost depends on requests (per million), request time, memory (GB) allocated to each function.</p> <p>First 1 million requests per month free - $0.20 per 1 million requests thereafter. $0.00001667 for every GB-second used.</p> <p>Additional AWS services imply cost (e.g. API Gateway, )</p> <p>Read more at https://aws.amazon.com/lambda/pricing/MongoDB Atlas Costs</p> <p>Cost depends on instance size, storage, iops, replication factor and backup retention.</p> <p>M0 free great for you (no VPC peering, use IP whitelist)M10 starts at $0.08/hr great for team DevM30 starts at $0.54.hr great for Production</p> <p>Read more at https://www.mongodb.com/cloud/atlas/pricing</p> <p>http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html</p> <p>Recurring ping hack serverless?</p> <p>Connections to MongoDB Atlas</p> <p>Encrypt using AWS KMS see this blog post: https://www.mongodb.com/blog/post/serverless-development-with-nodejs-aws-lambda-mongodb-atlas </p> <p>Container freeze &amp; recycle?Connection outside lambda function helpsOn scale new containers, new connectionsOk if API is used in bursts, but maybe not ok if used seldomly</p> <p>If not?</p> <p>Others?Local development? Lambda emulators</p> <p>python-lambda-local at https://pypi.python.org/pypi/python-lambda-local</p> <p>lambda-local (node.js) at https://www.npmjs.com/package/lambda-local </p> <p>Serverless frameworks evaluate them! F500s are!Serverless FrameworkZappaChaliceMore! - https://thenewstack.io/tns-guide-serverless-technologies-best-frameworks-platforms-tools/</p> <p>http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html</p> <p>Recurring ping hack serverless?</p> <p>Faade Serverless Functions logic querying backend API</p> <p>Backend Traditional stateful layer - CRUD API to Data Stores</p> <p>Would this be a Serverless Architecture ??Customer Single View - Insurance Industry (hypothetical) </p> <p>High-level architecture of a single view platform</p> <p>Stateful API Service Layer</p> <p>Note: Mention that Atlas, Lambda &amp; VPC have to be in the same region</p> <p>Note: Mention that Atlas, Lambda &amp; VPC have to be in the same region</p> <p>Serverless Architectures with AWS Lambda and MongoDB Atlas</p> <p>Q&amp;A</p> <p>Use code "Sig" for 25% off!Parties of 3+ get addtl 25%Sig NarvezSr. Solutions Architectsig@mongodb.com @SigNarvaez</p> <p>- Why MongoDB Atlas with AWS? Vs. native DynamoDB for example- Agg Fmwk Schema on-write revisit competitive analysis- &gt;&gt;&gt;&gt; Use an Agg Fmwk vs basic read for demo</p> <p>Performance optimization new area lambda evolving, so is better ways of handling connections to databasesDisadvantagesWhen to use, when no to - Lambda for simplicity and removing ops work but less performant, less control, Uber vs. Driving example eventual benefit vs. tco tipping pointCharge model</p> <p>Justify cost of DynamoDB vs MongoDB rabbit hole careful </p> <p>The key operational difference between FaaS and PaaS is scaling. With most PaaSs you still need to think about scale. Biggest advs: pay-for-use, no infra, </p>