Convert Your Code into a Microservice using AWS Lambda

Vyom NagraniSr. Product ManagerJuly 13th, 2015

Agenda

• What is AWS Lambda?

• Lambda common use cases

• How Lambda works

• Building a microservice on Lambda

• Hands-on exercises

• Q&A

What is AWS Lambda?

AWS Compute offerings

LambdaServerless compute platform for stateless

code execution in response to Triggers

ECSContainer

management service for running Docker on a managed cluster of

EC2

EC2Virtual servers in the Cloud

High performance at any scale; Cost-effective and

efficient

No Infrastructure to manage

Pay only for what you use: Lambda automatically matches capacity to your request rate. Purchase compute in 100ms increments.

Bring Your Own Code

“Productivity focused compute platform to build powerful, dynamic, modular applications in the cloud”

Run code in a choice of standard languages. Use threads, processes, files and shell scripts normally.

Focus on business logic, not infrastructure. You upload code; AWS Lambda handles everything else.

Why Lambda?

How Lambda works

S3 bucket notifications

DynamoDB Streams

Kinesis records

Cognito sync

SNS publish

Custom triggers

CloudTrail activity LambdaDynamoDB

Kinesis S3

Any custom

Invoked in response to triggers- Changes in data- Changes in state

Author in familiar language using any libraries; Execute only when needed, automatic scale

Redshift

SNS

Access any service, including your own

Any AWS

Such as…

“Lambda functions”

Old way of orchestrating workflows

SOURCE of data

Fleet of servers poll for changes

Listening to source Pushes to queue Pull off queue

Fleet of servers act as workers to process the

data

Auto-scale worker nodes to adjust with

load

S3 objectsSNS

messagesKinesis records

DDB TablesIoT Devices

…

Cross AZ replication,

Load Balancer

New way of orchestrating workflows

SOURCE of data

Attach a Lambda function

And that’s it!

Lambda does the listening, the polling,

the queuing, the autoscaling, and spins up as many

workers as needed to match the rate of change of source

data

Lambda common use cases

10

Typical usage scenarios

S3 + Lambda

Dynamic data ingestionImage thumbnailing, Video transcoding, File indexing, Log processing, Content validation, Aggregation and filtering

Kinesis + Lambda

Live stream processingApplication activity auditing, transaction order processing, Click stream analysis, IoT device response, telemetry and metering

SNS + Lambda

Smart IT, Custom messageAutomating IT alarm response, Custom actions, IT auditing, customizing broadcast messages, text to email push

Direct call + Lambda

Serverless backendMicroservices, Mobile backends, IoT backends

DynamoDB + Lambda

Database triggers[In preview] Data validation, Data filtering, Live notifications

Alexa + Lambda

Voice triggersBuild a custom Automated Voice Response system in the cloud

Typical workflow for dynamic data ingestion using S3+Lambda

Notification

Amazon S3 AWS Lambda processes the object

Amazon S3

New object uploaded

Amazon DynamoDB

Customers running dynamic data ingestion and processing using S3+Lambda

AWS Lambda

Indexing tables or

notifications

“I want to apply custom logic to process content being uploaded to my data store”. • Watermarking / thumbnailing• Transcoding• Indexing and deduplication• Aggregation and filtering• Pre processing• Content validation

Amazon S3 Bucket Events

Processed files

Typical workflow for real time stream processing using Kinesis+Lambda

Amazon Kinesis

AWS Lambda processes the stream

Amazon CloudWatch Logs

Amazon SNS

Customers running real-time data stream processing on Kinesis+Lambda

AWS Lambda

Aggregate statistics

Real-time analytics

Kinesis Stream

“I want to apply custom logic to process data being uploaded through my Kinesis stream”. • Client activity tracking• Metrics generation• Data cleansing• Log filtering• Indexing and searching• Log routing• Live alarms and notifications

Typical workflow for smart monitoring and automation using SNS+Lambda

AWS Lambda

Amazon SNS

Amazon CloudWatch Alarm

Typical workflow for custom messaging using SNS+Lambda

Amazon SNS

Amazon SNSAWS Lambda

How Lambda works

Lambda functions

Simple resource model• Set memory to any size from 128MB to 1GB, in 64MB steps• Receive an equivalent portion of other resources (disk, network, compute power,

etc.)• Lambda tells you how much memory you used, so you can tune this setting.

Flexible invocation paths• Lambda functions can be invoked “on demand” through CLI and Console• Subscribe to one or many event sources• Reuse the same Lambda function with multiple event sources

Granular permissions control (using IAM)• Define what permissions the function has• Uses IAM role (execution role) for granular permission control• Recommended minimum permission – log to CloudWatch• E.g. “read from <X> DDB table only in the context of <Y> function”

Writing and deploying Lambda Functions

• The Basics– Node.js or Java– AWS SDK comes built in and ready to use– Lambda handles inbound traffic– Use processes, threads, /tmp, sockets, …– Bring your own libraries, even native ones

• Deployment options– Author directly using the console– Package code as a ZIP and upload through uploadFunction API– 3rd Party plugins (Grunt, Jenkins, CloudBees, Codeship)

• Think stateless– Use S3, DynamoDB, or other Internet storage for persistent data– Don’t expect affinity to the infrastructure (you can’t “log in to the box”)

Monitoring and debugging Lambda Functions

• Console Dashboard• Lists all Lambda functions• Easy editing of resources, event

sources and other settings• At-a-glance metrics

• Metrics in CloudWatch• Requests• Errors• Latency• Throttles

• Logging in CloudWatch Logs

Invoking Lambda Functions

• Request Response: Call from mobile or web apps– Wait for a response– AWS SDK, AWS Mobile SDK, REST API, CLI

• Push Event: Incoming events from Amazon S3 or SNS– One event per Lambda invocation– Unordered model– 3 tries (won’t retry buggy code indefinitely)

• Pull Event: Get DynamoDB changes or Amazon Kinesis records as events– Ordered model with multiple records per event– Unlimited retries (until data expires)

Under the covers - Invocation permissions

• Resource policies – Used in the Push model– Define resource policies attached to a

Lambda function– E.g. “User X can invoke on function Y in the

context of bucket Z”– Resource policies allow for cross account

access!

• IAM roles– Used In the pull model– Lambda derives permission from execution

role to read from particular Stream– E.g. “User A has permissions to read from

Stream B in the context of Function C”

Building a microservice on Lambda

What are Microservices?

In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies.

-- James Lewis and Martin Fowler

http://martinfowler.com/microservices

What are Microservices? (Cont’d)

Microservices

Componentization via Smaller Services

Decentralized Governance

Organized around Business Capabilities

Smart endpoints and dumb pipes

Infrastructure Automation

Decentralized Data Management

Microservices and APIs

https://www.nginx.com/blog/building-microservices-using-an-api-gateway/

Sample Microservice on Lambda

https://aws.amazon.com/blogs/compute/the-squirrelbin-architecture-a-serverless-microservice-using-aws-lambda/

Hands-on exercises

Today’s exercises

1. Create and test your first Lambda function (Hello World)

2. Triggering Lambda from S3

3. Create a file deduplication Microservice– Use code from https://github.com/vyomnagrani/awslambda/blob/master/demo-s3-file-dedup.js

4. Triggering Lambda from SNS

5. Create a message customization Microservice– Use code from https://github.com/vyomnagrani/awslambda/blob/master/demo-sns-customize.js

6. Create a CRUD-backend Microservice with a public HTTP endpoint [Advanced users can jump directly to this one]

– Build off of https://github.com/vyomnagrani/awslambda/blob/master/demo-CRUD-backend.txt

Q&A

Visit http://aws.amazon.com/lambda, the AWS Compute blog, and the Lambda forum to learn more and get started using Lambda.

aws.amazon.com/activate

Everything and Anything Startups Need to Get Started on AWS