AWS Labmda x Python- Error Handling, Test and CI

2017-02-17

Takahiro Ikeuchieurie Inc.

AgendaOur use case of AWS Lambda

AWS Lambda and & Handling

AWS Lambda and Unit Test

2

Today's GoalLearn about error handlings in AWS Lambda

Learn about a local environment, UTand CI to develop AWS Lamba

3

Takahiro Ikeuchi @iktakahiroCompany / Community

eurie Inc. Founder & CEO

SQUEEZE Inc. Tech Adviser

PyData.Tokyo Organizer

Specialties (or just a dabbler :-D

Go lang, Python, React.js, TypeScript

Cloud Infrastructure, UI Design etc...

4

My Favorite AWS ServiceAmazon Aurora

5

eurie Inc.B2B SaaS (https://eurie.io/ja/)

Launched : Customer Support Solution for innovative team."eurie Desk"

Golang, React and Python (4:4:2)

AWS heavy user

6

Our use cases of AWS Lambda

7

Architecture

8

9

10

Use CasesIt is used as th hub of processing that allows async

POST documents to Elasticsearch

SMTP Service (e.g. Amazon SES, SendGrid)

Notifications to external system (e.g. Slack)

WebApp => SNS => AWS Lambda

When WebApp calls Amazon SNS, the response is returnedImmediately.

11

AWS Lambda and Error Handling

12

We don't know whether processing after SNS was successfulor not.

We must pray for safety :-D

When WebApp calls Amazon SNS, the response is returnedImmediately

“

“

13

Don't pray, Think about error handlings

14

Exeptions on AWS Lambda - 1(in async action)

If a process is failed, AWS Lambda retry automatically up to 2times.

When an exception is raised in Python, AWS determinesprocessing to be unsuccessful

def always_failed_handler(event, context): raise Exception('I failed!')

http://docs.aws.amazon.com/lambda/latest/dg/python-exceptions.html

15

Exeptions on AWS Lambda - 2The retry approach is not executed within a transaction.

Idempotency (冪統性) is needed.

1. Register a user data => Success

2. Send an email => Failure

When the 2nd process fails, the 1st process will also be retried.

Consider separating them into different AWS Lambda function.

16

Don't pray, Implement a retry logic

17

Exponential backoff

18

What is Exponential backoff

Exponential backoff - Wikipedia

Exponential backoff is an algorithm that uses feedback tomultiplicatively decrease the rate of some process, in order togradually find an acceptable rate.

“

“

19

Python and Exponential backoffhttps://github.com/rholder/retrying

You Just write @decorator.

my_function() is retried with the algorithm automatically when theexception is raised.

@retry(wait_exponential_multiplier=1000, stop_max_delay=12000, wait_exponential_max=1000)def my_functoin(): try: res = requests.post(URL, data=payload, timeout=5) except Exception: raise

20

Notes on using Exponential backoffIt is suitable for transient faults and system busy (500).

Retrying to a Bad Request response is a waste of time.

5 minute execution time limit, we can not retry unlimitedly.

21

NoteDead Letter Queuehttp://docs.aws.amazon.com/lambda/latest/dg/dlq.html

22

AWS Lambda and UT, CI

23

AWS Lambda and UnitTest - 1Debugging is slightly more difficult than general Linuxenvironment.

However, AWS Lambda calls the entry point which is just aPython function.

def lambda_handler(event, context): # main process

24

Write Test Code

25

AWS Lambda and UnitTest - 2dummy event and context object for Test

test_event = {"Records": [ { "EventVersion": "1.0", "EventSubscriptionArn": "arn:aws:sns:EXAMPLE", "EventSource": "aws:sns", "Sns": { "TopicArn": "arn:aws:sns:EXAMPLE", "Subject": "TestInvoke" } }]}

26

AWS Lambda and UnitTest - 3Write a test code to the entry point.

class TestLambdaFunction(object):

resp = lambda_function.lambda_handler(event, context) assert resp is True

Use a mock object as necessary.

27

Local EnviromentYou might adopt an emulator.

https://github.com/fugue/emulambda

difficult to implement the execution limit time of AWSLambda. The emulator has it.

28

Continuous Integration - 1

Archive the package for deployment include 3rd party packages.

pip install -U -r requirements.txt -t ./vendor

Vendoring fits this case.

├── lambda_function.py├── requirements.txt└── vendor ├── requests └── slackpy

Embedding a Python library in my own package - Stack Overflow

29

Continuous Integration - 2

We can adapt the knowledge we already got.

1. PUSH Github

2. Start CI Process (Codeship)

1. Build

2. Test

3. Deploy (aws cli)

aws lambda update-function-code --function-name my_func --zip-file src.zip

30

ConclusioinWe love Serverless Architecture!!

Knowledges about UnitTest and CI are adapt to AWS Lambda.

Write test code, not easy for human to notice the erroroccurred in the cloud.

Prepare a well-developed deployment processusing CI tool/service

31