2016 - Serverless Microservices on AWS with API Gateway and Lambda

Download 2016 - Serverless Microservices on AWS with API Gateway and Lambda

Post on 11-Jan-2017

693 views

Category:

Software

0 download

TRANSCRIPT

Building Microservices with API Gateway and Lambda

@mattjbarlowgithub.com/mattjbarlow

DevOps Days Austin 2016Revenge of the Devs

What we will talk aboutSwaggerAPI GatewayLambdaDynamoDB

What we will talk aboutSwaggerAPI GatewayLambdaDynamoDB

API Spec FileProxy / Router / URLBackend CodeData

With Swagger you can auto-generate...

DocumentationMonitoring TestsIntegration TestsClient Libraries

We Never Edit API Gateway Directly

We Edit Swaggerwhich is imported into...API GatewayMonitoringDocsTests

Example MicroserviceTeammates want the ability to run a Lambda Function exactly one time in the future.

SolutionAt Job MicroserviceA microservice that mimics the Unix at command for Lambda.

The At CommandFirst appears in 1979 as part of Unix Version 7.

Design

API MethodsPathOperationOperationIDDescription/atqGETlist_jobsList all jobs/atqPOSTcreate_jobCreate an at job/atq/{id}GETdescribe_jobDescribe an at job/atq/{id}DELETEdelete_jobDelete an at job

A method is a combination of a resource path and an operation.

ObjectsNameTypeFormatRequired?jobidstringuuidyeslambdaArnstringarnnotimestringdateTimeno

atJobThe object will define the response that our API returns.

CodeOperationIDPseudocodelist_jobsDynamoDB BatchGetItemdescribe_jobDynamoDB Query on jobidcreate_jobCreate CloudWatch Event and PutItem into DynamoDB.delete_jobDelete CloudWatch Event and DeleteItem out of DynamoDB.

These Operation IDs are defined in Swagger and passed through to Lambda code as part of the event object.

BatchGetItem can only return 100 items / 16KB of data, so paging is required.There is a limit of 50 Event Sources per AWS account.

Initialize Directory

git clone git@github.com:mattjbarlow/microservice-template.gitFileDescriptionservice/service.py *Python module that will run in Lambda.circle.ymlCircle CI instructions.deploy.ymlAnsible playbook for provisioning microservice.destroy.ymlAnsible playbook for deleting microservice.swagger.yml *Spec file that describes your API.template.jsonAWS resources required by the microservice.version.ymlAnsible playbook that versions your Lambda code.

* The bulk of your edits will be in these two files.

Update Swagger Paths

Remember these?PathOperationOperationIDDescription/atqGETlist_jobsList all jobs/atqPOSTcreate_jobCreate an at job/atq/{id}GETdescribe_jobDescribe an at job/atq/{id}DELETEdelete_jobDelete an at job

We Insert Them Directly Into Swagger

RESOURCE PATHHTTP OPERATIONObject Definition

Then we define our objects

Deploy

ansible-playbook -e prefix=devopsdays deploy.yml

Ansible Deployment PlaybookZips up Python module and uploads it to S3Creates the API GatewayProvisions AWS resourcesLambdaDynamoDBIAM Roles and PoliciesLambda PermissionAdds mapping templates to API GatewaySets stage variablesExports Swagger file from API Gateway

Import Swagger Into Postman

Open Postman

Click Import

Select Swagger File

Make sure you select the Swagger file exported by Ansible from AWS, not the swagger.yml that was downloaded from templates.

Start Sending Requests

Tail Lambda Logs

Logging in Lambda

API Gateway turned our HTTP request data into an Event Object. To start with, we log the entire Event Object.

pip install awslogs

awslogs allows us to mimic tail -f behavior on our Lambda functions Log Group.

Event Object Close-Up

Post BodyHeadersPath ParamsQuery Params

You dont HAVE to send all this information through to Lambda. You control what is sent to Lambda in the mapping template in Swagger.

API Gateway Variables

Helpful for filtering logs.

Tracing Our Request

GET Request LifecycleMethod ResponseIntegration ResponseLambdaMethod RequestIntegration RequestClientCloudwatch Logs

HTTP GET Request Is SentGET /dev/v1/atq HTTP/1.1Host: cg4e6xg82i.execute-api.us-east-1.amazonaws.comConnection: keep-aliveCache-Control: no-cacheUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36Postman-Token: 265fcfe1-e196-0114-89bf-442a34c0180dAccept: */*Accept-Encoding: gzip, deflate, sdchAccept-Language: en-US,en;q=0.8

Mapped To Event Object

Event Object Received by Lambda

Logged to CloudWatch

Editing Code

Dev Workflow

EditEvalPushRequestParse

create_job function

Only writes to Dynamo at this point. Later on we will create the CloudWatch event.

Push Code To Lambda

ansible-playbook -e prefix=dod version.yml

Test create_job

Important!

create_job response

Persisted in DynamoDB

POST Request LifecycleMethod ResponseIntegration ResponseLambdaMethod RequestIntegration RequestClientCloudwatch Logs

Events

Dynamo

Removing Lambda

GET Request With LambdaStepWhat HappensSend GET requestAPI Gateway Receives HTTP Request Data from client.TransformAPI Gateway Transforms Request Data into Event ObjectProxyAPI Gateway POSTs Event Object to LambdaReadLambda Reads data from DynamoDBReturnAPI Gateway Receives return values from LambdaTransformAPI Gateway Transforms backend data into HTTP ResponseRespondAPI Gateway Responds back to the client.

GET Request without LambdaStepWhat HappensSend GET requestAPI Gateway Receives HTTP Request Data from client.TransformAPI Gateway Transforms Request Data into Event ObjectProxyAPI Gateway POSTs DynamoDB Query to DynamoReadLambda Reads data from DynamoDBReturnAPI Gateway Receives return values from DynamoTransformAPI Gateway Transforms backend data into HTTP ResponseRespondAPI Gateway Responds back to the client.

Mapping GET Request To DynamoDB

Mapping DynamoDB Response to Client Response

Automating Tests

CI Workflow

CIStep 1Deploy MicroserviceStep 2Validate API CallsStep 3Destroy Microservice

git pushPOST: /project/:tree/:branch

Validating API Calls With Flexschema = load('swagger.awsexport.json')validate_api_call(schema, raw_request=r.request, raw_response=r)Receives Swagger spec file which is our source of truth.Makes an HTTP Request to the API Gateway URL.Ensures the response matches what we said it would in Swagger.

Loads Swagger spec into memory

AuthIt has been 1 days since we talked about auth.

Built In Auth OptionsGET /dev/v1/atq HTTP/1.1Host: cg4e6xg82i.execute-api.us-east-1.amazonaws.comConnection: keep-alivex-api-key: bkayZOMvuy8aZOhIgxq94K9Oe7Y70Hw55

Option 2: Signature Version 4 signing with IAM(Powerful, but requires client having AWS API Key)Option 1: API Keys managed by API Gateway API(Not really useful for user auth in public APIs)

You can use AWS credentials -- access and secret keys to sign requests to your service and authorize access like other AWS services. The signing of an Amazon API Gateway API request is managed by the custom API Gateway SDK generated for your service. You can retrieve temporary credentials associated with a role in your AWS account using Amazon Cognito.

Custom AuthorizersReceive Token from IdentitySend Token to /auth endpointPass ThroughResponds With JWTMake request with JWTCustom Authorizer InterceptCaches temporary policyAllows RequestGenerates JWTValidates JWTReturns temporary IAM policyClient LibraryAPI GatewayLambda$:

Validates Identity Token

Linkshttps://github.com/mattjbarlow/microservice-templatehttps://github.com/mattjbarlow/athttp://editor.swagger.io/#/http://flex-swagger.readthedocs.io/en/latest/