AWS Lambda at MindTouch

Download AWS Lambda at MindTouch

Post on 20-Mar-2017




1 download

Embed Size (px)


<ul><li><p>AWS LAMBDA @ .Juan Manuel Torres </p><p>Csar Lpez-Natarn Jan. 2016</p></li><li><p>JUAN AND CSAR</p><p>MindTouch Engineers</p></li><li><p>JUAN MANUEL TORRES</p><p> Born and raised in Bogot Colombia </p><p> Software Engineer @ </p><p> Love </p><p> Programming </p><p> Photography </p><p> Lizards </p><p> Traveling </p><p></p></li><li><p>CSAR LPEZ-NATARN</p><p> Senior Software Engineer @ </p><p> Love to work on: </p><p> APIs </p><p> Data engineering </p><p> Making code fast </p><p> Infrastructure automation </p><p></p></li><li><p>Customer Success SaaS</p></li><li><p> Authoring and publishing platform for Product Help </p><p> We use your products knowledge to: </p><p> Accelerate User Adoption </p><p> Build Customer Loyalty </p><p> We help you: </p><p> Understand your Customers </p><p> Accelerate your Sales Cycle</p></li><li><p>SOME MINDTOUCH CUSTOMERS</p></li><li><p>MINDTOUCHS SCALE</p><p> Up to 2M req/day </p><p> Up to 5M events/day</p></li><li><p>AWS LAMBDAServerless computing</p></li><li><p>WHAT IS AWS LAMBDA?</p><p> Compute service that hosts an execution environment for your code on high availability compute infrastructure. </p><p> The service: </p><p> Administers the compute resources (CPU, memory, network and OS maintenance). </p><p> Guarantees the selected capacity. </p><p> Scales your software automatically. </p><p> Provides monitoring and logging services.</p></li><li><p>WHY DOES IT MATTER?</p><p> Because all the tasks described before consume programmers time </p><p> And instead of focusing on your problem domain you are dealing with those other things </p><p> It is actually complicated to get all those pieces right</p></li><li><p>HOW DO I USE IT?</p><p> You provide your Node.js, Java, or Python code. </p><p> Configure: </p><p> Event sources </p><p> Memory </p><p> Execution role </p><p> Handler function</p></li><li><p>LETS GET ONE PYTHON LAMBDA FUNCTION DEPLOYED</p><p>photo by Kool Cats Photography</p><p></p></li><li><p>DEMOLambda S3 to Tweet</p><p></p></li><li><p>LAMBDA FUNCTION LIFE CYCLE</p><p> Develop Test Deploy Repeat</p></li><li><p>LAMBDA FUNCTION DEVELOPMENTphoto by Lachlan Hardy</p><p></p></li><li><p>PROGRAMMING MODEL / ENTRY POINT</p><p>Handler </p><p> 1 def lambda_handler(event, context): 2 s3_info = event['Records'][0]['s3'] 3 ... 4 tweet_images.send_image( 5 s3_info['bucket']['name'], 6 s3_info['object']['key'], 7 cleanup=True 8 )</p></li><li><p>PROGRAMMING MODEL / LOGGING</p><p>Logging </p><p> 1 import logging 2 3 logger = logging.getLogger() 4 logger.setLevel(logging.INFO) 5 6 def lambda_handler(event, context): 7"Invoked with {0} events".format( 8 len(event['Records'])))</p></li><li><p>LAMBDA FUNCTION TESTING</p></li><li><p>TESTING OPTIONS</p><p> Local testing through unit testing </p><p> Make sure you can swap out implementations of components that perform I/O </p><p> Mock testing through the AWS Console in the Lambda service itself </p><p> Create your own simulation environment for Lambda</p></li><li><p>photo by NASA's Marshall Space Flight Center</p><p>LAMBDA FUNCTION DEPLOYMENT</p><p></p></li><li><p>DEPLOYMENT</p><p> Creating the lambda function on the service </p><p> Granting permissions to event sources and any other components that the functions needs to interact with </p><p> Sending the packaged code to the service </p><p> Configuring the application</p></li><li><p>AWS LAMBDA USAGE @ MINDTOUCH</p><p> Mirror data streams across accounts and regions </p><p> Archive events to S3 </p><p> Generate SQS messages from S3 object creation </p><p> Publish Auto Scaling Groups notifications to DataDog</p></li><li><p>CHALLENGES FACEDwhile using AWS Lambda</p></li><li><p>CHALLENGES WHILE USING AWS LAMBDA</p><p> Consistent function configuration </p><p> Automated function deployment on different environments (dev, test, production)</p></li><li><p>MINDTOUCH LAMBDA OPSMindTouch deployment tool for Lambda functions </p></li><li><p>MINDTOUCH LAMBDA OPS DESIGN OBJECTIVE</p><p>To enable a consistent development/test/deployment lifecycle for AWS Lambda functions.</p></li><li><p>TOOL REQUIREMENTS</p><p> Developers + DevOps </p><p> Configuration must be simple </p><p> Single file </p><p> Checked-in to version control </p><p> Many deployments </p><p> Permission management </p><p> Managing Event Source Mappings</p></li><li><p>IMPLEMENTATION DETAILS</p><p> Single YAML Configuration file </p><p> Automatic generation of config.json including names for all sources and resources </p><p> Prepares lambda function for deployment </p><p> Creates a CloudFormation stack </p></li><li><p>DEMOKinesis to S3</p></li><li><p>KINESIS2S3XMLARCHIVER CONFIGURATION</p><p> 1 app: 2 3 resources: 4 source: 5 type: kinesis 6 name: events-stream 7 event-source: 8 batch-size: 10000 9 starting-position: latest 10 11 s3bucket: 12 type: s3 13 name: events-archive 14 allow: 15 - s3:PutObject 16 - s3:GetObject</p></li><li><p>photo by Andreas Levers</p><p>WARNINGS</p><p></p></li><li><p>LIMITS</p><p> Execution time is limited to 300s (5 minutes) </p><p> 1.5GB memory limits</p><p>photo by Navalarp Teratanatorn</p><p></p></li><li><p>SECURITY</p><p> As Jan 2016 there is no support for accessing resources on a VPC (Coming soon)</p></li><li><p>DEPLOYMENT</p><p> No straight forward deployment other than the console or using a third party tool</p><p>photo by NASA's Marshall Space Flight Center</p><p></p></li><li><p>CONCLUSIONS Huge developer productivity </p><p>gains Reduces complexity in large </p><p>infrastructure systems They do what they claim to do We will choose AWS Lambda </p><p>for suitable projects in the future</p></li><li><p>MINDTOUCH IS HIRING!</p><p>Join the Nerds</p><p></p></li></ul>