Transcript
Page 1: AWS Lambda at MindTouch

AWS LAMBDA @ .Juan Manuel Torres

César López-Natarén Jan. 2016

Page 2: AWS Lambda at MindTouch

JUAN AND CÉSAR

MindTouch Engineers

Page 3: AWS Lambda at MindTouch

JUAN MANUEL TORRES

➤ Born and raised in Bogotá Colombia

➤ Software Engineer @

➤ Love

➤ Programming

➤ Photography

➤ Lizards

➤ Traveling

[email protected]

Page 4: AWS Lambda at MindTouch

CÉSAR LÓPEZ-NATARÉN

➤ Senior Software Engineer @

➤ Love to work on:

➤ APIs

➤ Data engineering

➤ Making code fast

➤ Infrastructure automation

[email protected]

Page 5: AWS Lambda at MindTouch

Customer Success SaaS

Page 6: AWS Lambda at MindTouch

➤ Authoring and publishing platform for Product Help

➤ We use your product’s knowledge to:

➤ Accelerate User Adoption

➤ Build Customer Loyalty

➤ We help you:

➤ Understand your Customers

➤ Accelerate your Sales Cycle

Page 7: AWS Lambda at MindTouch

SOME MINDTOUCH CUSTOMERS

Page 8: AWS Lambda at MindTouch

MINDTOUCH’S SCALE

➤ Up to 2M req/day

➤ Up to 5M events/day

Page 9: AWS Lambda at MindTouch

AWS LAMBDAServerless computing

Page 10: AWS Lambda at MindTouch

WHAT IS AWS LAMBDA?

➤ Compute service that hosts an execution environment for your code on high availability compute infrastructure.

➤ The service:

➤ Administers the compute resources (CPU, memory, network and OS maintenance).

➤ Guarantees the selected capacity.

➤ Scales your software automatically.

➤ Provides monitoring and logging services.

Page 11: AWS Lambda at MindTouch

WHY DOES IT MATTER?

➤ Because all the tasks described before consume programmer’s time

➤ And instead of focusing on your problem domain you are dealing with those other things

➤ It is actually complicated to get all those pieces right

Page 12: AWS Lambda at MindTouch

HOW DO I USE IT?

➤ You provide your Node.js, Java, or Python code.

➤ Configure:

➤ Event sources

➤ Memory

➤ Execution role

➤ Handler function

Page 14: AWS Lambda at MindTouch

DEMOLambda S3 to Tweet

Page 15: AWS Lambda at MindTouch

LAMBDA FUNCTION LIFE CYCLE

➤ Develop ➤ Test ➤ Deploy ➤ Repeat

Page 17: AWS Lambda at MindTouch

PROGRAMMING MODEL / ENTRY POINT

Handler

1 def lambda_handler(event, context): 2 s3_info = event['Records'][0]['s3'] 3 ... 4 tweet_images.send_image( 5 s3_info['bucket']['name'], 6 s3_info['object']['key'], 7 cleanup=True 8 )

Page 18: AWS Lambda at MindTouch

PROGRAMMING MODEL / LOGGING

Logging

1 import logging 2 3 logger = logging.getLogger() 4 logger.setLevel(logging.INFO) 5 6 def lambda_handler(event, context): 7 logger.info("Invoked with {0} events".format( 8 len(event['Records'])))

Page 19: AWS Lambda at MindTouch

LAMBDA FUNCTION TESTING

Page 20: AWS Lambda at MindTouch

TESTING OPTIONS

➤ Local testing through unit testing

➤ Make sure you can swap out implementations of components that perform I/O

➤ Mock testing through the AWS Console in the Lambda service itself

➤ Create your own simulation environment for Lambda

Page 22: AWS Lambda at MindTouch

DEPLOYMENT

➤ Creating the lambda function on the service

➤ Granting permissions to event sources and any other components that the functions needs to interact with

➤ Sending the packaged code to the service

➤ Configuring the application

Page 23: AWS Lambda at MindTouch

AWS LAMBDA USAGE @ MINDTOUCH

➤ Mirror data streams across accounts and regions

➤ Archive events to S3

➤ Generate SQS messages from S3 object creation

➤ Publish Auto Scaling Group’s notifications to DataDog

Page 24: AWS Lambda at MindTouch

CHALLENGES FACEDwhile using AWS Lambda

Page 25: AWS Lambda at MindTouch

CHALLENGES WHILE USING AWS LAMBDA

➤ Consistent function configuration

➤ Automated function deployment on different environments (dev, test, production)

Page 26: AWS Lambda at MindTouch

MINDTOUCH LAMBDA OPSMindTouch deployment tool for Lambda functions

Page 27: AWS Lambda at MindTouch

MINDTOUCH LAMBDA OPS DESIGN OBJECTIVE

To enable a consistent development/test/deployment lifecycle for AWS Lambda functions.

Page 28: AWS Lambda at MindTouch

TOOL REQUIREMENTS

➤ Developers + DevOps

➤ Configuration must be simple

• Single file

• Checked-in to version control

• Many deployments

➤ Permission management

➤ Managing Event Source Mappings

Page 29: AWS Lambda at MindTouch

IMPLEMENTATION DETAILS

➤ Single YAML Configuration file

➤ Automatic generation of config.json including names for all sources and resources

➤ Prepares lambda function for deployment

➤ Creates a CloudFormation stack

Page 30: AWS Lambda at MindTouch

DEMOKinesis to S3

Page 31: AWS Lambda at MindTouch

KINESIS2S3XMLARCHIVER CONFIGURATION

1 app: 2 3 resources: 4 source: 5 type: kinesis 6 name: events-stream 7 event-source: 8 batch-size: 10000 9 starting-position: latest 10 11 s3bucket: 12 type: s3 13 name: events-archive 14 allow: 15 - s3:PutObject 16 - s3:GetObject

Page 33: AWS Lambda at MindTouch

LIMITS

➤ Execution time is limited to 300s (5 minutes)

➤ 1.5GB memory limits

photo by Navalarp Teratanatorn

Page 34: AWS Lambda at MindTouch

SECURITY

➤ As Jan 2016 there is no support for accessing resources on a VPC (Coming soon)

Page 36: AWS Lambda at MindTouch

CONCLUSIONS ➤ Huge developer productivity

gains ➤ Reduces complexity in large

infrastructure systems ➤ They do what they claim to do ➤ We will choose AWS Lambda

for suitable projects in the future

Page 37: AWS Lambda at MindTouch

MINDTOUCH IS HIRING!www.mindtouch.com/jobs

Join the Nerds


Top Related