aws meetup building_lambda

28
Building Lambda Packages In Python Getting Past Hello World

Upload: adam-book

Post on 14-Apr-2017

122 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Aws meetup building_lambda

Building Lambda Packages In Python

Getting Past Hello World

Page 2: Aws meetup building_lambda

Sponsors

Page 3: Aws meetup building_lambda

Find me on LinkedIn

AWS Certifications

Presented by Adam Book

Page 4: Aws meetup building_lambda

Lambda Uses

Using a serverless technology like AWS Lambda takes a bit of shift in thinking If you think about the quote from Werner Vogles:

“Everything fails all the time”

Then you might currently have an AutoScaling group of min 1 max 1 Desired 1 just for an instance that is processing a piece of data or doing a task on a daily or scheduled basis.

Lambda may be the cost effective way to take over that event.

Page 5: Aws meetup building_lambda

Lambda Uses

Using a serverless technology like AWS Lambda takes a bit of shift in thinking Event Based Processing

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html

AWS Services as Event Sources S3 – Push ModelDynamo DB – Pull Model Kinesis – Pull Model Simple Notification Service – Push Model Amazon Cognito – Push Model

CloudWatchLogs - Push model

CloudWatch Events – Push Model

Scheduled Events (cloudwatch) – Push Model

AWS Config – Push model

Page 6: Aws meetup building_lambda

Cascading Lambda Functions

One Lambda Events can be the triggering Event for another

Dynamo DB

redshift

S3

Page 7: Aws meetup building_lambda

Ways to Invoke Lambda

Currently there are 2 ways to invoke Lambda

EventThis invocation type causes AWS Lambda to execute the Lambda function asynchronously.The event sources Amazon S3, Amazon SNS or Amazon DynamoDB use this invocation type.

Request Response This invocation type causes AWS Lambda to execute the function synchronously and returns the response immediately to the calling application. This invocation type is available for custom applications.

Page 8: Aws meetup building_lambda

Cases where Lambdaisn’t a good fit

The AWS Certified Solutions Architect – Associate Level exam is intended for individuals who perform a Solutions Architect role.

Scenario

The need for the latest OS patches and Security updates minutes or hours after they’re releasedThe need to access the Security logs the host

The need to change the OS settings such as Memory Allocation or max file handlers An auditing and compliance requirement that requires you to take a snapshot of a host after a security issue

Page 9: Aws meetup building_lambda

Cases where Lambdais a good Fit

The AWS Certified Solutions Architect – Associate Level exam is intended for individuals who perform a Solutions Architect role.

Scenario

Automatically Tagging newly Created Instances

Automatically Shutting off

Performing Database functions such as bulk inserts and vacuums

Converting Bash Scripts which run automation tasks

Sending notifications to your Slack Channel

Page 10: Aws meetup building_lambda

Lambda Pricing

AWS Lambda is one of the most Economical Services on AWS

Requests• First 1 million requests per month are free• $0.20 per 1 million requests thereafter ($0.0000002 per request)

If all of this sounds confusing then you might try this calculator found on Matthew Fuller’s blog

For more infohttps://aws.amazon.com/lambda/pricing/

Page 11: Aws meetup building_lambda

Programming Model forLambda Functions in Python

In building your Python Lambda packages there are 4 main topics to know :

• The Lambda Function Handler• The Context Object• Logging• Exceptions

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html

NOTE: At the time of presentation you must write your Python lambda code in 2.7

Page 12: Aws meetup building_lambda

The Handler

The Handler is the function AWS Lambda calls to start execution of your Lambda Function.

When a Lambda function is invoked then it starts executing your code by calling the handler function.

It also passes any event data as the first parameter.

The Handler is the function AWS Lambda calls to start execution of your Lambda Function.

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html

Page 13: Aws meetup building_lambda

The Handler

def lambda_handler(event, context): myfile = retrieveFile(input_file, bucket) newfile = converter(myfile,index,mytype) return success

The Handler is the function AWS Lambda calls to start execution of your Lambda Function.

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html

Page 14: Aws meetup building_lambda

Logging (in Python)

The following Python statements generate log entries:• print statements.• Logger functions in the logging module ( ie logging.Logger.info /

logging.Logger.errror)

Both print and logging.* function write logs to CloudWatch logs but the logging.* functions write additional information to each log entry such as timestamp and log level.

The Handler is the function AWS Lambda calls to start execution of your Lambda Function.

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/python-logging.html

Page 15: Aws meetup building_lambda

Time to Test Testing your Lambda

Image by http://www.splitshire.com/

Page 16: Aws meetup building_lambda

Lambda Testing Checklist

Is the function timeout to low?

Does the function have the correct IAM Role and permissions for services that it’s using and to send logs to Cloudwatch Logs?

Has the function been allocated enough memory?

Page 17: Aws meetup building_lambda

Building the Package

Step 1 Start with a small / micro EC2 instance running AWS linux

Once you’ve written your code (And Tested it) now you can build your package with dependencies, zip it up and then upload.

NOTE: There’s other ways to build the package…this is the fastest most effective way I’ve found for python

Page 18: Aws meetup building_lambda

Building the Packagecont.

Step 2 SSH to your instance and create a directory for your Lambda code

NOTE: The name of the file doesn’t matter as much as the lambda handler calling your other functions does.

Step 3 Move your code to that directory either via scp or by cutting and pasting to a file with a .py extension

$mkdir /home/aws-user/cleaner_lambda

Page 19: Aws meetup building_lambda

Building the PackageAdding Dependencies

Step 4 Install any libaries or dependencies that you need via pip directly to the directory which you created.

NOTE: The name of the file doesn’t matter as much as the lambda handler calling your other functions does.

$pip install requests –t /home/aws-user/cleaner_lambda

Page 20: Aws meetup building_lambda

Building the PackageCreating the zip file

Step 5 Create the zip file and add NOT ONLY your python file but also the directories of the packages which pip has downloaded for you

NOTE: using the –r flag will recursively add all the files in the subsequent folders (in our case requests)

$zip –r lambda_cleanup.zip lambda_cleanup.py requests/

Page 21: Aws meetup building_lambda

Python Lambda Packages What about writing to files?

Image by http://www.gratisography.com/

Page 22: Aws meetup building_lambda

Writing to files

def retrieveFile(filename, bucket): s3conn = boto.connect_s3() bucket = s3conn.get_bucket(bucket) fn=filename key = bucket.get_key(fn) dl = key.get_contents_to_filename('/tmp/'+fn) myfile = ("/tmp/"+fn) return myfile

You can write to files (while processing) but you must do so in /tmp

After you have manipulated your file then you will need to save somewhere else, most likely an S3 bucket

Page 23: Aws meetup building_lambda

Which Packages don’t need to be added

There are a handful of libraries that when imported do not need to be built into a zip file.

• Boto / Boto3 • Time• OS

For more infohttp://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html

Page 24: Aws meetup building_lambda

Python Lambda Packages Real World Examples

Image by http://www.gratisography.com/

Page 26: Aws meetup building_lambda

Lambda in other Flavors

Python 2.7 Java 8node.js 4.3

node.js 0.10

Page 27: Aws meetup building_lambda

Questions?

Image by http://www.gratisography.com/

Page 28: Aws meetup building_lambda

Interested in SponsoringAWS Atlanta?

Image by http://www.gratisography.com/

Contact us at

http://www.meetup.com/AWS-Atlanta/