aws meetup building_lambda
TRANSCRIPT
Building Lambda Packages In Python
Getting Past Hello World
Sponsors
Find me on LinkedIn
AWS Certifications
Presented by Adam Book
Lambda Uses
Using a serverless technology like AWS Lambda takes a bit of shift in thinking If you think about the quote from Werner Vogles:
“Everything fails all the time”
Then you might currently have an AutoScaling group of min 1 max 1 Desired 1 just for an instance that is processing a piece of data or doing a task on a daily or scheduled basis.
Lambda may be the cost effective way to take over that event.
Lambda Uses
Using a serverless technology like AWS Lambda takes a bit of shift in thinking Event Based Processing
For more infohttp://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html
AWS Services as Event Sources S3 – Push ModelDynamo DB – Pull Model Kinesis – Pull Model Simple Notification Service – Push Model Amazon Cognito – Push Model
CloudWatchLogs - Push model
CloudWatch Events – Push Model
Scheduled Events (cloudwatch) – Push Model
AWS Config – Push model
Cascading Lambda Functions
One Lambda Events can be the triggering Event for another
Dynamo DB
redshift
S3
Ways to Invoke Lambda
Currently there are 2 ways to invoke Lambda
EventThis invocation type causes AWS Lambda to execute the Lambda function asynchronously.The event sources Amazon S3, Amazon SNS or Amazon DynamoDB use this invocation type.
Request Response This invocation type causes AWS Lambda to execute the function synchronously and returns the response immediately to the calling application. This invocation type is available for custom applications.
Cases where Lambdaisn’t a good fit
The AWS Certified Solutions Architect – Associate Level exam is intended for individuals who perform a Solutions Architect role.
Scenario
The need for the latest OS patches and Security updates minutes or hours after they’re releasedThe need to access the Security logs the host
The need to change the OS settings such as Memory Allocation or max file handlers An auditing and compliance requirement that requires you to take a snapshot of a host after a security issue
Cases where Lambdais a good Fit
The AWS Certified Solutions Architect – Associate Level exam is intended for individuals who perform a Solutions Architect role.
Scenario
Automatically Tagging newly Created Instances
Automatically Shutting off
Performing Database functions such as bulk inserts and vacuums
Converting Bash Scripts which run automation tasks
Sending notifications to your Slack Channel
Lambda Pricing
AWS Lambda is one of the most Economical Services on AWS
Requests• First 1 million requests per month are free• $0.20 per 1 million requests thereafter ($0.0000002 per request)
If all of this sounds confusing then you might try this calculator found on Matthew Fuller’s blog
For more infohttps://aws.amazon.com/lambda/pricing/
Programming Model forLambda Functions in Python
In building your Python Lambda packages there are 4 main topics to know :
• The Lambda Function Handler• The Context Object• Logging• Exceptions
For more infohttp://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html
NOTE: At the time of presentation you must write your Python lambda code in 2.7
The Handler
The Handler is the function AWS Lambda calls to start execution of your Lambda Function.
When a Lambda function is invoked then it starts executing your code by calling the handler function.
It also passes any event data as the first parameter.
The Handler is the function AWS Lambda calls to start execution of your Lambda Function.
For more infohttp://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html
The Handler
def lambda_handler(event, context): myfile = retrieveFile(input_file, bucket) newfile = converter(myfile,index,mytype) return success
The Handler is the function AWS Lambda calls to start execution of your Lambda Function.
For more infohttp://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html
Logging (in Python)
The following Python statements generate log entries:• print statements.• Logger functions in the logging module ( ie logging.Logger.info /
logging.Logger.errror)
Both print and logging.* function write logs to CloudWatch logs but the logging.* functions write additional information to each log entry such as timestamp and log level.
The Handler is the function AWS Lambda calls to start execution of your Lambda Function.
For more infohttp://docs.aws.amazon.com/lambda/latest/dg/python-logging.html
Time to Test Testing your Lambda
Image by http://www.splitshire.com/
Lambda Testing Checklist
Is the function timeout to low?
Does the function have the correct IAM Role and permissions for services that it’s using and to send logs to Cloudwatch Logs?
Has the function been allocated enough memory?
Building the Package
Step 1 Start with a small / micro EC2 instance running AWS linux
Once you’ve written your code (And Tested it) now you can build your package with dependencies, zip it up and then upload.
NOTE: There’s other ways to build the package…this is the fastest most effective way I’ve found for python
Building the Packagecont.
Step 2 SSH to your instance and create a directory for your Lambda code
NOTE: The name of the file doesn’t matter as much as the lambda handler calling your other functions does.
Step 3 Move your code to that directory either via scp or by cutting and pasting to a file with a .py extension
$mkdir /home/aws-user/cleaner_lambda
Building the PackageAdding Dependencies
Step 4 Install any libaries or dependencies that you need via pip directly to the directory which you created.
NOTE: The name of the file doesn’t matter as much as the lambda handler calling your other functions does.
$pip install requests –t /home/aws-user/cleaner_lambda
Building the PackageCreating the zip file
Step 5 Create the zip file and add NOT ONLY your python file but also the directories of the packages which pip has downloaded for you
NOTE: using the –r flag will recursively add all the files in the subsequent folders (in our case requests)
$zip –r lambda_cleanup.zip lambda_cleanup.py requests/
Python Lambda Packages What about writing to files?
Image by http://www.gratisography.com/
Writing to files
def retrieveFile(filename, bucket): s3conn = boto.connect_s3() bucket = s3conn.get_bucket(bucket) fn=filename key = bucket.get_key(fn) dl = key.get_contents_to_filename('/tmp/'+fn) myfile = ("/tmp/"+fn) return myfile
You can write to files (while processing) but you must do so in /tmp
After you have manipulated your file then you will need to save somewhere else, most likely an S3 bucket
Which Packages don’t need to be added
There are a handful of libraries that when imported do not need to be built into a zip file.
• Boto / Boto3 • Time• OS
For more infohttp://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html
Python Lambda Packages Real World Examples
Image by http://www.gratisography.com/
Python Lambda Packages Real World Examples
A Cloudwatch to Slack Lambda Function
A Lambda Job to backup RDS instances
A Lambda Job that cleans S3 buckets
Word Analysis with Lambda
Nightly Shutdown (and startup) with Lambda
Lambda in other Flavors
Python 2.7 Java 8node.js 4.3
node.js 0.10
Questions?
Image by http://www.gratisography.com/
Interested in SponsoringAWS Atlanta?
Image by http://www.gratisography.com/
Contact us at
http://www.meetup.com/AWS-Atlanta/