aws lambda at mindtouch

Download AWS Lambda at MindTouch

Post on 20-Mar-2017

933 views

Category:

Technology

1 download

Embed Size (px)

TRANSCRIPT

  • AWS LAMBDA @ .Juan Manuel Torres

    Csar Lpez-Natarn Jan. 2016

  • JUAN AND CSAR

    MindTouch Engineers

  • JUAN MANUEL TORRES

    Born and raised in Bogot Colombia

    Software Engineer @

    Love

    Programming

    Photography

    Lizards

    Traveling

    juant@mindtouch.com

  • CSAR LPEZ-NATARN

    Senior Software Engineer @

    Love to work on:

    APIs

    Data engineering

    Making code fast

    Infrastructure automation

    cesarn@mindtouch.com

  • Customer Success SaaS

  • Authoring and publishing platform for Product Help

    We use your products knowledge to:

    Accelerate User Adoption

    Build Customer Loyalty

    We help you:

    Understand your Customers

    Accelerate your Sales Cycle

  • SOME MINDTOUCH CUSTOMERS

  • MINDTOUCHS SCALE

    Up to 2M req/day

    Up to 5M events/day

  • AWS LAMBDAServerless computing

  • WHAT IS AWS LAMBDA?

    Compute service that hosts an execution environment for your code on high availability compute infrastructure.

    The service:

    Administers the compute resources (CPU, memory, network and OS maintenance).

    Guarantees the selected capacity.

    Scales your software automatically.

    Provides monitoring and logging services.

  • WHY DOES IT MATTER?

    Because all the tasks described before consume programmers time

    And instead of focusing on your problem domain you are dealing with those other things

    It is actually complicated to get all those pieces right

  • HOW DO I USE IT?

    You provide your Node.js, Java, or Python code.

    Configure:

    Event sources

    Memory

    Execution role

    Handler function

  • LETS GET ONE PYTHON LAMBDA FUNCTION DEPLOYED

    photo by Kool Cats Photography

    https://www.flickr.com/photos/katsrcool/8441623385/in/photolist-dRXwwM-6rzaq-vaxEt-iYjNi-shZR8H-BP6Ths-hfG25t-r8rTU9-NWe1M-63ouQ2-7yqTAS-492vhe-ek2yUc-iYjNu-2vqaDs-g1n2Jc-Bh6ADY-qt1PFS-5XRau8-8woteL-7Uz3bf-exhg4K-gr3HLK-buGxMr-87cX83-4EbwuK-5oawmX-ouziGt-bxTFf8-czEcR9-dyhByU-rq1pbv-m8uMCQ-f46qXy-5vRSeE-iDtLD-9h3ZRR-d9SWy4-r6G3H2-7LApXP-rq1rJM-bU5J1k-bFaY6C-bU5Hkx-9bYe8B-5Afais-4F3PQ5-sjgRW-85hW1D-66gtnk

  • DEMOLambda S3 to Tweet

    http://github.com/onema/lambda-tweet

  • LAMBDA FUNCTION LIFE CYCLE

    Develop Test Deploy Repeat

  • LAMBDA FUNCTION DEVELOPMENTphoto by Lachlan Hardy

    https://www.flickr.com/photos/lachlanhardy/4150836513/in/photolist-7jN85M-89ZXVw-dAxJkf-6aLKEK-4T92Mm-6aLLQK-4T92R7-4T4Pic-4T92Sw-4T4PoT-8TouHU-c49FWS-7w2Qiv-8ToBjb-3EoBAZ-aLQ4oz-chYQ3d-aLKF52-aLPKYe-aLJZ7a-aLRjgx-aLKHEF-aLM9yF-aLKdEn-aLNxpB-aLK6Qk-aLK3Rx-aLJrTD-aLHJJT-aLKaAz-aLJKA2-aLNJiZ-aLLzAt-aLPPVD-aLNB68-aLM3ga-aLJVw4-aLPxjZ-aLKwjB-aLNRcF-aLPGec-aLMSqx-aLHUBt-aLNYwx-aLKoq6-aLNiKv-aLHRhr-aLNqo4-aLJdn2-aLJ9BK

  • PROGRAMMING MODEL / ENTRY POINT

    Handler

    1 def lambda_handler(event, context): 2 s3_info = event['Records'][0]['s3'] 3 ... 4 tweet_images.send_image( 5 s3_info['bucket']['name'], 6 s3_info['object']['key'], 7 cleanup=True 8 )

  • PROGRAMMING MODEL / LOGGING

    Logging

    1 import logging 2 3 logger = logging.getLogger() 4 logger.setLevel(logging.INFO) 5 6 def lambda_handler(event, context): 7 logger.info("Invoked with {0} events".format( 8 len(event['Records'])))

  • LAMBDA FUNCTION TESTING

  • TESTING OPTIONS

    Local testing through unit testing

    Make sure you can swap out implementations of components that perform I/O

    Mock testing through the AWS Console in the Lambda service itself

    Create your own simulation environment for Lambda

  • photo by NASA's Marshall Space Flight Center

    LAMBDA FUNCTION DEPLOYMENT

    https://www.flickr.com/photos/nasamarshall/10537112073/in/photolist-h48sj6-buxcTH-7Bzhy5-6Zgz2v-6Zkzim-6Zgzhn-6Zkzdd-6Zgz6K-7Bvs7v-7BvrDn-hQA3kk-hQzavg-zaMCRR-6ZfAxv-zayZaL-7BvrPX-7Bvszc-7Bzh6s-7Bvsg4-7BzgaG-7BvtfR-7Bvt2p-7BzhpJ-7BzgS9-7BzfGC-7BvsXX-7Bzgp9-7BzfVY-7Bzhbh-7BvtHP-7BvsNi-7Bzfqj-7BvsJg-7BzhCs-7BvsDp-7BzfLb-7Bzgjs-7Bzhu3-7BzhjW-9muzn6-uUWUjU-9Si4EV-yVfXSE-beA8Jv-vAZhz3-yUSN3d-zabwXu-zabwnb-zdjZt6-zabMsb

  • DEPLOYMENT

    Creating the lambda function on the service

    Granting permissions to event sources and any other components that the functions needs to interact with

    Sending the packaged code to the service

    Configuring the application

  • AWS LAMBDA USAGE @ MINDTOUCH

    Mirror data streams across accounts and regions

    Archive events to S3

    Generate SQS messages from S3 object creation

    Publish Auto Scaling Groups notifications to DataDog

  • CHALLENGES FACEDwhile using AWS Lambda

  • CHALLENGES WHILE USING AWS LAMBDA

    Consistent function configuration

    Automated function deployment on different environments (dev, test, production)

  • MINDTOUCH LAMBDA OPSMindTouch deployment tool for Lambda functions

  • MINDTOUCH LAMBDA OPS DESIGN OBJECTIVE

    To enable a consistent development/test/deployment lifecycle for AWS Lambda functions.

  • TOOL REQUIREMENTS

    Developers + DevOps

    Configuration must be simple

    Single file

    Checked-in to version control

    Many deployments

    Permission management

    Managing Event Source Mappings

  • IMPLEMENTATION DETAILS

    Single YAML Configuration file

    Automatic generation of config.json including names for all sources and resources

    Prepares lambda function for deployment

    Creates a CloudFormation stack

  • DEMOKinesis to S3

  • KINESIS2S3XMLARCHIVER CONFIGURATION

    1 app: 2 3 resources: 4 source: 5 type: kinesis 6 name: events-stream 7 event-source: 8 batch-size: 10000 9 starting-position: latest 10 11 s3bucket: 12 type: s3 13 name: events-archive 14 allow: 15 - s3:PutObject 16 - s3:GetObject

  • photo by Andreas Levers

    WARNINGS

    https://www.flickr.com/photos/96dpi/3816391053/in/photolist-6Pf16K-xoo1Ex-7Jr58C-6hyJfm-gi7dDE-27j4Fz-rFHiUC-715c6E-5hgcgY-4bDYnT-5aDvTF-9m1RFF-4wnDfX-3PWAyZ-oM5hzK-uryPfz-q9V65t-6t9zJq-xqazZ2-2qcbYf-9gQTWb-u3PSC-JhxDh-8E9VN8-MWqZU-mQTGa-MWrrG-gVZH-RZgPP-8iXPV7-ftFc35-7QLcMm-7sx9M-nd13Wy-axZCgx-E12h4-cZFkY-8M8MQU-27CZY-7RtVy-2p7uj-531s9c-8aArhd-FF7Bi-LkcM-jhJMoe-rHUVBv-qMdpKH-rHTUGC-rrrtXd

  • LIMITS

    Execution time is limited to 300s (5 minutes)

    1.5GB memory limits

    photo by Navalarp Teratanatorn

    https://500px.com/photo/57874328/sky-is-the-limit-by-navalarp-teratanatorn

  • SECURITY

    As Jan 2016 there is no support for accessing resources on a VPC (Coming soon)

  • DEPLOYMENT

    No straight forward deployment other than the console or using a third party tool

    photo by NASA's Marshall Space Flight Center

    https://www.flickr.com/photos/nasamarshall/10537112073/in/photolist-h48sj6-buxcTH-7Bzhy5-6Zgz2v-6Zkzim-6Zgzhn-6Zkzdd-6Zgz6K-7Bvs7v-7BvrDn-hQA3kk-hQzavg-zaMCRR-6ZfAxv-zayZaL-7BvrPX-7Bvszc-7Bzh6s-7Bvsg4-7BzgaG-7BvtfR-7Bvt2p-7BzhpJ-7BzgS9-7BzfGC-7BvsXX-7Bzgp9-7BzfVY-7Bzhbh-7BvtHP-7BvsNi-7Bzfqj-7BvsJg-7BzhCs-7BvsDp-7BzfLb-7Bzgjs-7Bzhu3-7BzhjW-9muzn6-uUWUjU-9Si4EV-yVfXSE-beA8Jv-vAZhz3-yUSN3d-zabwXu-zabwnb-zdjZt6-zabMsb

  • CONCLUSIONS Huge developer productivity

    gains Reduces complexity in large

    infrastructure systems They do what they claim to do We will choose AWS Lambda

    for suitable projects in the future

  • MINDTOUCH IS HIRING!www.mindtouch.com/jobs

    Join the Nerds

    http://www.mindtouch.com/jobs