aws meetup building_lambda

Building Lambda Packages In Python

Getting Past Hello World

Sponsors

Find me on LinkedIn

AWS Certifications

Presented by Adam Book

Lambda Uses

Using a serverless technology like AWS Lambda takes a bit of shift in thinking If you think about the quote from Werner Vogles:

“Everything fails all the time”

Then you might currently have an AutoScaling group of min 1 max 1 Desired 1 just for an instance that is processing a piece of data or doing a task on a daily or scheduled basis.

Lambda may be the cost effective way to take over that event.

Lambda Uses

Using a serverless technology like AWS Lambda takes a bit of shift in thinking Event Based Processing

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html

AWS Services as Event Sources S3 – Push ModelDynamo DB – Pull Model Kinesis – Pull Model Simple Notification Service – Push Model Amazon Cognito – Push Model

CloudWatchLogs - Push model

CloudWatch Events – Push Model

Scheduled Events (cloudwatch) – Push Model

AWS Config – Push model

Cascading Lambda Functions

One Lambda Events can be the triggering Event for another

Dynamo DB

redshift

S3

Ways to Invoke Lambda

Currently there are 2 ways to invoke Lambda

EventThis invocation type causes AWS Lambda to execute the Lambda function asynchronously.The event sources Amazon S3, Amazon SNS or Amazon DynamoDB use this invocation type.

Request Response This invocation type causes AWS Lambda to execute the function synchronously and returns the response immediately to the calling application. This invocation type is available for custom applications.

Cases where Lambdaisn’t a good fit

The AWS Certified Solutions Architect – Associate Level exam is intended for individuals who perform a Solutions Architect role.

Scenario

The need for the latest OS patches and Security updates minutes or hours after they’re releasedThe need to access the Security logs the host

The need to change the OS settings such as Memory Allocation or max file handlers An auditing and compliance requirement that requires you to take a snapshot of a host after a security issue

Cases where Lambdais a good Fit

The AWS Certified Solutions Architect – Associate Level exam is intended for individuals who perform a Solutions Architect role.

Scenario

Automatically Tagging newly Created Instances

Automatically Shutting off

Performing Database functions such as bulk inserts and vacuums

Converting Bash Scripts which run automation tasks

Sending notifications to your Slack Channel

Lambda Pricing

AWS Lambda is one of the most Economical Services on AWS

Requests• First 1 million requests per month are free• $0.20 per 1 million requests thereafter ($0.0000002 per request)

If all of this sounds confusing then you might try this calculator found on Matthew Fuller’s blog

For more infohttps://aws.amazon.com/lambda/pricing/

Programming Model forLambda Functions in Python

In building your Python Lambda packages there are 4 main topics to know :

• The Lambda Function Handler• The Context Object• Logging• Exceptions

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html

NOTE: At the time of presentation you must write your Python lambda code in 2.7

The Handler

The Handler is the function AWS Lambda calls to start execution of your Lambda Function.

When a Lambda function is invoked then it starts executing your code by calling the handler function.

It also passes any event data as the first parameter.

The Handler is the function AWS Lambda calls to start execution of your Lambda Function.

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html

The Handler

def lambda_handler(event, context): myfile = retrieveFile(input_file, bucket) newfile = converter(myfile,index,mytype) return success

The Handler is the function AWS Lambda calls to start execution of your Lambda Function.

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html

Logging (in Python)

The following Python statements generate log entries:• print statements.• Logger functions in the logging module ( ie logging.Logger.info /

logging.Logger.errror)

Both print and logging.* function write logs to CloudWatch logs but the logging.* functions write additional information to each log entry such as timestamp and log level.

The Handler is the function AWS Lambda calls to start execution of your Lambda Function.

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/python-logging.html

Time to Test Testing your Lambda

Image by http://www.splitshire.com/

Lambda Testing Checklist

Is the function timeout to low?

Does the function have the correct IAM Role and permissions for services that it’s using and to send logs to Cloudwatch Logs?

Has the function been allocated enough memory?

Building the Package

Step 1 Start with a small / micro EC2 instance running AWS linux

Once you’ve written your code (And Tested it) now you can build your package with dependencies, zip it up and then upload.

NOTE: There’s other ways to build the package…this is the fastest most effective way I’ve found for python

Building the Packagecont.

Step 2 SSH to your instance and create a directory for your Lambda code

NOTE: The name of the file doesn’t matter as much as the lambda handler calling your other functions does.

Step 3 Move your code to that directory either via scp or by cutting and pasting to a file with a .py extension

$mkdir /home/aws-user/cleaner_lambda

Building the PackageAdding Dependencies

Step 4 Install any libaries or dependencies that you need via pip directly to the directory which you created.

NOTE: The name of the file doesn’t matter as much as the lambda handler calling your other functions does.

$pip install requests –t /home/aws-user/cleaner_lambda

Building the PackageCreating the zip file

Step 5 Create the zip file and add NOT ONLY your python file but also the directories of the packages which pip has downloaded for you

NOTE: using the –r flag will recursively add all the files in the subsequent folders (in our case requests)

$zip –r lambda_cleanup.zip lambda_cleanup.py requests/

Python Lambda Packages What about writing to files?

Image by http://www.gratisography.com/

Writing to files

def retrieveFile(filename, bucket): s3conn = boto.connect_s3() bucket = s3conn.get_bucket(bucket) fn=filename key = bucket.get_key(fn) dl = key.get_contents_to_filename('/tmp/'+fn) myfile = ("/tmp/"+fn) return myfile

You can write to files (while processing) but you must do so in /tmp

After you have manipulated your file then you will need to save somewhere else, most likely an S3 bucket

Which Packages don’t need to be added

There are a handful of libraries that when imported do not need to be built into a zip file.

• Boto / Boto3 • Time• OS

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html

Python Lambda Packages Real World Examples

Image by http://www.gratisography.com/

Python Lambda Packages Real World Examples

A Cloudwatch to Slack Lambda Function

A Lambda Job to backup RDS instances

A Lambda Job that cleans S3 buckets

Word Analysis with Lambda

Nightly Shutdown (and startup) with Lambda

Lambda in other Flavors

Python 2.7 Java 8node.js 4.3

node.js 0.10

Questions?

Image by http://www.gratisography.com/

Interested in SponsoringAWS Atlanta?

Image by http://www.gratisography.com/

Contact us at

http://www.meetup.com/AWS-Atlanta/