AWS Lambda at MindTouch

Download AWS Lambda at MindTouch

Post on 20-Mar-2017

932 views

Category:

Technology

1 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>AWS LAMBDA @ .Juan Manuel Torres </p><p>Csar Lpez-Natarn Jan. 2016</p></li><li><p>JUAN AND CSAR</p><p>MindTouch Engineers</p></li><li><p>JUAN MANUEL TORRES</p><p> Born and raised in Bogot Colombia </p><p> Software Engineer @ </p><p> Love </p><p> Programming </p><p> Photography </p><p> Lizards </p><p> Traveling </p><p> juant@mindtouch.com</p></li><li><p>CSAR LPEZ-NATARN</p><p> Senior Software Engineer @ </p><p> Love to work on: </p><p> APIs </p><p> Data engineering </p><p> Making code fast </p><p> Infrastructure automation </p><p> cesarn@mindtouch.com</p></li><li><p>Customer Success SaaS</p></li><li><p> Authoring and publishing platform for Product Help </p><p> We use your products knowledge to: </p><p> Accelerate User Adoption </p><p> Build Customer Loyalty </p><p> We help you: </p><p> Understand your Customers </p><p> Accelerate your Sales Cycle</p></li><li><p>SOME MINDTOUCH CUSTOMERS</p></li><li><p>MINDTOUCHS SCALE</p><p> Up to 2M req/day </p><p> Up to 5M events/day</p></li><li><p>AWS LAMBDAServerless computing</p></li><li><p>WHAT IS AWS LAMBDA?</p><p> Compute service that hosts an execution environment for your code on high availability compute infrastructure. </p><p> The service: </p><p> Administers the compute resources (CPU, memory, network and OS maintenance). </p><p> Guarantees the selected capacity. </p><p> Scales your software automatically. </p><p> Provides monitoring and logging services.</p></li><li><p>WHY DOES IT MATTER?</p><p> Because all the tasks described before consume programmers time </p><p> And instead of focusing on your problem domain you are dealing with those other things </p><p> It is actually complicated to get all those pieces right</p></li><li><p>HOW DO I USE IT?</p><p> You provide your Node.js, Java, or Python code. </p><p> Configure: </p><p> Event sources </p><p> Memory </p><p> Execution role </p><p> Handler function</p></li><li><p>LETS GET ONE PYTHON LAMBDA FUNCTION DEPLOYED</p><p>photo by Kool Cats Photography</p><p>https://www.flickr.com/photos/katsrcool/8441623385/in/photolist-dRXwwM-6rzaq-vaxEt-iYjNi-shZR8H-BP6Ths-hfG25t-r8rTU9-NWe1M-63ouQ2-7yqTAS-492vhe-ek2yUc-iYjNu-2vqaDs-g1n2Jc-Bh6ADY-qt1PFS-5XRau8-8woteL-7Uz3bf-exhg4K-gr3HLK-buGxMr-87cX83-4EbwuK-5oawmX-ouziGt-bxTFf8-czEcR9-dyhByU-rq1pbv-m8uMCQ-f46qXy-5vRSeE-iDtLD-9h3ZRR-d9SWy4-r6G3H2-7LApXP-rq1rJM-bU5J1k-bFaY6C-bU5Hkx-9bYe8B-5Afais-4F3PQ5-sjgRW-85hW1D-66gtnk</p></li><li><p>DEMOLambda S3 to Tweet</p><p>http://github.com/onema/lambda-tweet</p></li><li><p>LAMBDA FUNCTION LIFE CYCLE</p><p> Develop Test Deploy Repeat</p></li><li><p>LAMBDA FUNCTION DEVELOPMENTphoto by Lachlan Hardy</p><p>https://www.flickr.com/photos/lachlanhardy/4150836513/in/photolist-7jN85M-89ZXVw-dAxJkf-6aLKEK-4T92Mm-6aLLQK-4T92R7-4T4Pic-4T92Sw-4T4PoT-8TouHU-c49FWS-7w2Qiv-8ToBjb-3EoBAZ-aLQ4oz-chYQ3d-aLKF52-aLPKYe-aLJZ7a-aLRjgx-aLKHEF-aLM9yF-aLKdEn-aLNxpB-aLK6Qk-aLK3Rx-aLJrTD-aLHJJT-aLKaAz-aLJKA2-aLNJiZ-aLLzAt-aLPPVD-aLNB68-aLM3ga-aLJVw4-aLPxjZ-aLKwjB-aLNRcF-aLPGec-aLMSqx-aLHUBt-aLNYwx-aLKoq6-aLNiKv-aLHRhr-aLNqo4-aLJdn2-aLJ9BK</p></li><li><p>PROGRAMMING MODEL / ENTRY POINT</p><p>Handler </p><p> 1 def lambda_handler(event, context): 2 s3_info = event['Records'][0]['s3'] 3 ... 4 tweet_images.send_image( 5 s3_info['bucket']['name'], 6 s3_info['object']['key'], 7 cleanup=True 8 )</p></li><li><p>PROGRAMMING MODEL / LOGGING</p><p>Logging </p><p> 1 import logging 2 3 logger = logging.getLogger() 4 logger.setLevel(logging.INFO) 5 6 def lambda_handler(event, context): 7 logger.info("Invoked with {0} events".format( 8 len(event['Records'])))</p></li><li><p>LAMBDA FUNCTION TESTING</p></li><li><p>TESTING OPTIONS</p><p> Local testing through unit testing </p><p> Make sure you can swap out implementations of components that perform I/O </p><p> Mock testing through the AWS Console in the Lambda service itself </p><p> Create your own simulation environment for Lambda</p></li><li><p>photo by NASA's Marshall Space Flight Center</p><p>LAMBDA FUNCTION DEPLOYMENT</p><p>https://www.flickr.com/photos/nasamarshall/10537112073/in/photolist-h48sj6-buxcTH-7Bzhy5-6Zgz2v-6Zkzim-6Zgzhn-6Zkzdd-6Zgz6K-7Bvs7v-7BvrDn-hQA3kk-hQzavg-zaMCRR-6ZfAxv-zayZaL-7BvrPX-7Bvszc-7Bzh6s-7Bvsg4-7BzgaG-7BvtfR-7Bvt2p-7BzhpJ-7BzgS9-7BzfGC-7BvsXX-7Bzgp9-7BzfVY-7Bzhbh-7BvtHP-7BvsNi-7Bzfqj-7BvsJg-7BzhCs-7BvsDp-7BzfLb-7Bzgjs-7Bzhu3-7BzhjW-9muzn6-uUWUjU-9Si4EV-yVfXSE-beA8Jv-vAZhz3-yUSN3d-zabwXu-zabwnb-zdjZt6-zabMsb</p></li><li><p>DEPLOYMENT</p><p> Creating the lambda function on the service </p><p> Granting permissions to event sources and any other components that the functions needs to interact with </p><p> Sending the packaged code to the service </p><p> Configuring the application</p></li><li><p>AWS LAMBDA USAGE @ MINDTOUCH</p><p> Mirror data streams across accounts and regions </p><p> Archive events to S3 </p><p> Generate SQS messages from S3 object creation </p><p> Publish Auto Scaling Groups notifications to DataDog</p></li><li><p>CHALLENGES FACEDwhile using AWS Lambda</p></li><li><p>CHALLENGES WHILE USING AWS LAMBDA</p><p> Consistent function configuration </p><p> Automated function deployment on different environments (dev, test, production)</p></li><li><p>MINDTOUCH LAMBDA OPSMindTouch deployment tool for Lambda functions </p></li><li><p>MINDTOUCH LAMBDA OPS DESIGN OBJECTIVE</p><p>To enable a consistent development/test/deployment lifecycle for AWS Lambda functions.</p></li><li><p>TOOL REQUIREMENTS</p><p> Developers + DevOps </p><p> Configuration must be simple </p><p> Single file </p><p> Checked-in to version control </p><p> Many deployments </p><p> Permission management </p><p> Managing Event Source Mappings</p></li><li><p>IMPLEMENTATION DETAILS</p><p> Single YAML Configuration file </p><p> Automatic generation of config.json including names for all sources and resources </p><p> Prepares lambda function for deployment </p><p> Creates a CloudFormation stack </p></li><li><p>DEMOKinesis to S3</p></li><li><p>KINESIS2S3XMLARCHIVER CONFIGURATION</p><p> 1 app: 2 3 resources: 4 source: 5 type: kinesis 6 name: events-stream 7 event-source: 8 batch-size: 10000 9 starting-position: latest 10 11 s3bucket: 12 type: s3 13 name: events-archive 14 allow: 15 - s3:PutObject 16 - s3:GetObject</p></li><li><p>photo by Andreas Levers</p><p>WARNINGS</p><p>https://www.flickr.com/photos/96dpi/3816391053/in/photolist-6Pf16K-xoo1Ex-7Jr58C-6hyJfm-gi7dDE-27j4Fz-rFHiUC-715c6E-5hgcgY-4bDYnT-5aDvTF-9m1RFF-4wnDfX-3PWAyZ-oM5hzK-uryPfz-q9V65t-6t9zJq-xqazZ2-2qcbYf-9gQTWb-u3PSC-JhxDh-8E9VN8-MWqZU-mQTGa-MWrrG-gVZH-RZgPP-8iXPV7-ftFc35-7QLcMm-7sx9M-nd13Wy-axZCgx-E12h4-cZFkY-8M8MQU-27CZY-7RtVy-2p7uj-531s9c-8aArhd-FF7Bi-LkcM-jhJMoe-rHUVBv-qMdpKH-rHTUGC-rrrtXd</p></li><li><p>LIMITS</p><p> Execution time is limited to 300s (5 minutes) </p><p> 1.5GB memory limits</p><p>photo by Navalarp Teratanatorn</p><p>https://500px.com/photo/57874328/sky-is-the-limit-by-navalarp-teratanatorn</p></li><li><p>SECURITY</p><p> As Jan 2016 there is no support for accessing resources on a VPC (Coming soon)</p></li><li><p>DEPLOYMENT</p><p> No straight forward deployment other than the console or using a third party tool</p><p>photo by NASA's Marshall Space Flight Center</p><p>https://www.flickr.com/photos/nasamarshall/10537112073/in/photolist-h48sj6-buxcTH-7Bzhy5-6Zgz2v-6Zkzim-6Zgzhn-6Zkzdd-6Zgz6K-7Bvs7v-7BvrDn-hQA3kk-hQzavg-zaMCRR-6ZfAxv-zayZaL-7BvrPX-7Bvszc-7Bzh6s-7Bvsg4-7BzgaG-7BvtfR-7Bvt2p-7BzhpJ-7BzgS9-7BzfGC-7BvsXX-7Bzgp9-7BzfVY-7Bzhbh-7BvtHP-7BvsNi-7Bzfqj-7BvsJg-7BzhCs-7BvsDp-7BzfLb-7Bzgjs-7Bzhu3-7BzhjW-9muzn6-uUWUjU-9Si4EV-yVfXSE-beA8Jv-vAZhz3-yUSN3d-zabwXu-zabwnb-zdjZt6-zabMsb</p></li><li><p>CONCLUSIONS Huge developer productivity </p><p>gains Reduces complexity in large </p><p>infrastructure systems They do what they claim to do We will choose AWS Lambda </p><p>for suitable projects in the future</p></li><li><p>MINDTOUCH IS HIRING!www.mindtouch.com/jobs</p><p>Join the Nerds</p><p>http://www.mindtouch.com/jobs</p></li></ul>