@awscloud es designing next-gen apps: serverless€¦ · building blocks for serverless...

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Ignacio García @igngar_cloud

Designing next-gen apps: ServerlessOctober 2018

AWS Solutions Architect

@awscloud_es


Agenda

• Introduction to Serverless

• Use case: Microservices

• Viesgo: Datalake

• Recap - resources


Serverless means…

F l ex i b l e S ca l i n g

N o I d l e C a p a c i t y

$

H i g h Ava i l a b i l i t y

N o S e r v e r M a n a g e m e n t


Serverless Growth

Hundreds of thousands of active customers

From students building Alexa Skills to Enterprise.

3X year-over-year usage growth

Vibrant startup ecosystem

stdlib, IOpipe, Serverless.com, FaunaDB


Building blocks for serverless applications

AWS Lambda Amazon DynamoDB

Amazon SNS

Amazon API Gateway Amazon SQS Amazon Kinesis

Amazon S3

Orchestration and State Management

API Proxy and GraphQL Messaging and Queues Analytics

Monitoring and Debugging

Compute Storage Database

AWS X-RayAWS Step Functions Amazon Cognito

User Management and IdP

AWS AppSync Amazon Athena

AWS Lambda@Edge Amazon Aurora Serverless


A typical day for a developer…

Manager: “We need an app to let our customers send feedback– our competitors just launched the same thing, so I need it fast. I don’t want to pay a lot for it, especially when no one is using it. But remember that we’re growing, so make sure it scales great and is easy to manage and operate. And you’re on your own – sorry!”

Developer “Not a problem. I’ll make it serverless…”


A typical day for a Serverless developer…

Serverless web app with

…an API ???

…access to existing data stored in ???

…static content served by ???

…dynamic content/business logic encoded as ???

… authenticate/authorize application ???

… orchestrate application logic ???

Joe promised h is boss an app.Now what?

Step 1: Find out what to use!

Joe we needs


Amazon API Gateway

Create a unified

API frontend for

multiple micro-

services

Authenticate and

authorize requests

to a backend

DDoS protection

and throttling for

your backend

Throttle, meter,

and monetize API

usage by third-

party developers


Building an API with Amazon API Gateway

Internet

Mobile Apps

Websites

Services

AWS Lambda functions

API Gateway Cache

Endpoints on Amazon EC2

All publicly accessible endpoints

Amazon CloudWatch Monitoring

Amazon CloudFront

Any other AWS service

Endpoints on Amazon VPC

CognitoAuthorizer

CustomAuthorizer

API Authorization

PrivateAPI endpoints


API Configuration

You can create APIs

Define resources within an API

Define methods for a resource

• Methods are Resource + HTTP verb

Pet Store

/pets

/pets/{petId}

• GET

• POST

• PUT

API Configuration can be deployed to a

stage

Stages are different environments

For example:

• Dev (e.g. awsapigateway.com/dev)

• Beta (e.g. awsapigateway.com/beta)

• Prod (e.g. awsapigateway.com/prod)

• As many stages as you need

Pet Store

dev

beta

gamma

prodCanary releases within a stage!


AWS API Gateway customers

“With Amazon API Gateway and AWS Lambda, the user

experience is up to 90% faster. That's for both

photographers uploading images and the editorial team processing them.”

Marco ViganòHead of Digital Development

Using a common API layer powered by Amazon API

Gateway and Network Load Balancer, Fox can completely decouple the frontend of FOX

NOW from the backend content system.

NH uses API Gateway for their NH PCI Bubble workload.


Business logic encoded in AWS Lambda

Changes in data state

Requests to endpoints

Changes in resource state

EVENT SOURCE SERVICES (ANYTHING)FUNCTION

Node.jsPythonJavaC#GolangPowershell

SLA: 99.95% Monthly Uptime Percentage

https://aws.amazon.com/lambda/sla/

Up to 15 min


Lambda execution models

Asynchronous (event) Poll-based

Amazon SNS

AWS Lambda function

Amazon S3

reqs

AmazonAPI Gateway

AWS Lambda function

/order

Synchronous (push)

Amazon DynamoDB

Amazon Kinesis

changes

AWS Lambda service

function

Amazon SQS


Tweak your function’s computer power

Lambda exposes only a memory control, with the % of CPU core and network capacity allocated to a function proportionally

Is your code CPU, Network or memory-bound? If so, it could be cheaper to choose more memory.


Concise function logic

• Separate Lambda handler (entry point) from core logic

• Use functions to TRANSFORM, not TRANSPORT

• Dynamic logic via configuration• Per function – Environment variables

• Cross function – Amazon Parameter Store/Secrets Manager

• Read only what you need. For example:• Properly indexed databases

• Query filters in Aurora

• Use S3 select


AWS Lambda customers

Seamlessly ramps up to peak traffic of

22K concurrent requests

Quick time to market: 8 weeks from conception to production

Ableto scale up to 20,000 concurrent Lambda executions in

testing

One simulation of 20 million mortgages ran in 1.5 hours, more

than 4X faster than existing process

Enel uses AWS Lambda for their Factelec workload to generate

electronic bills


ExtractImageMetadata

ImageTypeCheck

TransformMetadata

Rekognition Thumbnail

StoreMetadata

Start

End

AWS Step Functions

• Execute One or One Million• Scales out• Doesn’t lose state• Deals with errors/timeouts• Auditable

• Easy to build! • Defined in JSON – see your progress in the console.


AWS Step Functions

State Types

Task: a single unit of work “I want to sequence functions”

Retry: If fails, retry “I want to retry functions”

Choice: a single unit of work “I want to select functions based on data”

Parallel: fork and join data across tasks “I want to run functions in parallel”

Pass: passes inputs to its outputs ”I want to pass this till the next step”

Wait: wait until n seconds. “I have code that runs for hours”

Fail: Stops and mark as failure “I want try/catch/finally”

Success: Stops an execution successfully ”I finished!”

ExtractImageMetadata

ImageTypeCheck

TransformMetadata

Rekognition Thumbnail

StoreMetadata

Start

End


No orchestration in code

STA

RT

JOB

JOB

#X

STA

RTE

D

HT

TP P

OST

HT

TP P

OST

AR

E W

E TH

ERE

YET?

NO

PE!

WE’

RE

DO

NE!

ZzZz

OR

time.sleep(10)


No orchestration in code – use AWS Step Functions!


Step Functions customers

Custom monitoring and alarming with step functions.

“Developers do not have to be experts in workflow, and the

developers who wrote the Lambda function can easily create the Step

Functions workflow. We can put more developers on projects, and that will

ultimately enable the creation of more internal solutions that will

drive efficiency and productivity. ”

Paul BrownSenior Developer Manager

Coca-Cola uses Step Functions for:

Vending pass worklow

Nutrition information validation workflow


Accessing stored data in Amazon DynamoDB

Dead Simple• GetItem(primaryKey)

• PutItem(item)

Amazon DynamoDB Accelerator

Amazon DynamoDB

Applications

Amazon DynamoDB - StreamsMore than a hundred thousand AWS customers use Amazon DynamoDB!


Amazon Dynamo DB customers

“We recently wrote a zero-infrastructure service using AWS Lambda and Amazon

DynamoDB,”

Capital One uses DynamoDB to reduce latency for their

mobile applications by moving their mainframe transactions

to a serverless architecture for unbound scale.

Samsung Electronics uses DynamoDB for their petabyte

size mobile app backups, resulting in consistent high

performance and cost savings.


Serving static content with S3 and Cloudfront

Amazon CloudFront Amazon S3


Managing Access with Cognito

Web and Mobile Apps

Amazon Cognito

Developers focus on what is special about

their app

Cognito handles auth and identity

Federation

Managed User Directory

Hosted UI

AWS Credentials

Standard Tokens


Bringing all that together…

CloudFront

Content Delivery

ApplicationLayer

PersistencyLayer

S3

Static Content

APILayer

API GatewayDynamoDB

AWS Lambda

Cognito

AWS Step Functions

AWS Step Functions


CloudFormation templateAWSTemplateFormatVersion: '2010-09-09'Resources:

GetHtmlFunctionGetHtmlPermissionProd:Type: AWS::Lambda::PermissionProperties:Action: lambda:invokeFunctionPrincipal: apigateway.amazonaws.comFunctionName:

Ref: GetHtmlFunctionSourceArn:

Fn::Sub: arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${ServerlessRestApi}/Prod/ANY/*ServerlessRestApiProdStage:

Type: AWS::ApiGateway::StageProperties:DeploymentId:

Ref: ServerlessRestApiDeploymentRestApiId:

Ref: ServerlessRestApiStageName: Prod

ListTable:Type: AWS::DynamoDB::TableProperties:ProvisionedThroughput:

WriteCapacityUnits: 5ReadCapacityUnits: 5

AttributeDefinitions:- AttributeName: id

AttributeType: SKeySchema:- KeyType: HASH

AttributeName: idGetHtmlFunction:

Type: AWS::Lambda::FunctionProperties:Handler: index.gethtmlCode:

S3Bucket: flourish-demo-bucketS3Key: todo_list.zip

Role:Fn::GetAtt:- GetHtmlFunctionRole- Arn

Runtime: nodejs4.3GetHtmlFunctionRole:

Type: AWS::IAM::RoleProperties:ManagedPolicyArns:- arn:aws:iam::aws:policy/AmazonDynamoDBReadOnlyAccess

- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRoleAssumeRolePolicyDocument:

Version: '2012-10-17'Statement:- Action:

- sts:AssumeRoleEffect: AllowPrincipal:

Service:- lambda.amazonaws.com

ServerlessRestApiDeployment:Type: AWS::ApiGateway::DeploymentProperties:RestApiId:

Ref: ServerlessRestApiDescription: 'RestApi deployment id: 127e3fb91142ab1ddc5f5446adb094442581a90d'StageName: Stage

GetHtmlFunctionGetHtmlPermissionTest:Type: AWS::Lambda::PermissionProperties:Action: lambda:invokeFunctionPrincipal: apigateway.amazonaws.comFunctionName:

Ref: GetHtmlFunctionSourceArn:

Fn::Sub: arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${ServerlessRestApi}/*/ANY/*ServerlessRestApi:

Type: AWS::ApiGateway::RestApiProperties:Body:

info:version: '1.0'title:

Ref: AWS::StackNamepaths:

"/{proxy+}":x-amazon-apigateway-any-method:x-amazon-apigateway-integration:

httpMethod: ANYtype: aws_proxyuri:

Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${GetHtmlFunction.Arn}/invocationsresponses: {}

swagger: '2.0'


AWS Serverless Application Model (SAM)

CloudFormation extension optimized for serverless

New serverless resource types: functions, APIs, and tables

Supports anything CloudFormation supports

Open specification (Apache 2.0)github.com/awslabsserverless-application-model


SAM template

AWSTemplateFormatVersion: "2010-09-09"Transform: AWS::Serverless-2016-10-31

Resources: GetHtmlFunction:Type: AWS::Serverless::FunctionProperties:

CodeUri: s3://sam-demo-bucket/todo_list.zipHandler: index.gethtmlRuntime: nodejs4.3Policies: AmazonDynamoDBReadOnlyAccessEvents:GetHtml:

Type: ApiProperties:

Path: /{proxy+}Method: ANY

ListTable:Type: AWS::Serverless::SimpleTable

Tells CloudFormation this is a SAM template it needs to “transform”

Creates a Lambda function with the referenced managed IAM policy, runtime, code at the referenced zip location, and handler as defined. Also creates an API Gateway and takes care of all mapping/permissions necessary

Creates a DynamoDB table with 5 Read & Write units


Meet SAM CLI – Local API Gateway/Lambda

CLI tool for local testing of serverless apps

Works with Lambda functions and “proxy-style” APIs

Response object and function logs available on your local machine

Supports all native runtimes

github.com/awslabs/aws-sam-cli

R u n L o c a l l y

D e b u g L o c a l l y

Nuevos Modelos de Negocio apalancados en la tecnología. Gestor de servicios energéticos integral

Generación distribuida en BTClientes finales o

pequeñas plantas de energía limpia

P2PEconomía colaborativa también en el

sector eléctrico

AlmacenamientoLa energía almacenada se convierte en energía

eléctrica cuando sea necesario

Coche eléctricoNo solo un sistema de movilidad alternativo, sino sistemas de

almacenamiento móvil que pueden interactuar con la red

Edificios automatizadosSistemas energéticamente eficientes que actúan

directamente en el edificio y brindan flexibilidad a la red

Virtual energy plantsTecnología de generación

distribuida con un potencial de crecimiento significativo

Smart homesHogares equipados con

dispositivos electrónicos (por ejemplo, iluminación, calefacción) que se pueden controlar de forma

remota

Relaciones con clienteLos minoristas totalmente digitales ofrecen una experiencia

completamente nueva a sus clientes

BlockchainUso potencial de la tecnología en toda la cadena

de valor

Porque un DataLake……..y porque Serverless

Reducción del time to

market no servidores que

gestionar

Escala bajo demanda 24 horas pero no se paga

cuando no esta funcionandoDisponibilidad y

tolerancia al fallo

Menor Equipo de operación

Flujo de datos analitico……..entran datos y sales respuestas que se convierten en datos nuevamente

Ingesta

Orquestar y Tranformar

DATOS

RESPUESTAS

Almacena Analiza y Procesa

VisualizarConsumir

SERVERLESS

Flujo de datos analítico Ingesta y Almacena…….. como entran los datos y se persisten en el lake

SchemaDefinido

por el usuario

Organización del Lake

Nuestro S3 como “source of truth”

• Landing: Donde llegan la información en crudo• Trasnformed : Donde se almacena transformada• Enriched Donde se enriquece la información tras procesos

de transformación

Landing y transforma están gobernado por el catalogo de Glue

Notificación

• Eventos de cargas Ok o no OK

Tópico denotificaciones

Flujo de datos analítico Analiza y Procesa……..como se procesan , se analizan y enriquecen

Amazon EC2

Amazon

EMR

Op

cio

ne

s d

ep

en

die

nd

o

de

la n

ec

es

ida

d

Analizar

• Athena con objetivo exploratorios y autoservicio• Redshift para nuestro procesos y reporting formal y

corporativo

Orquestación y transformación

• Step funciones como orquestador de la transformación.

• Varias posibilidades para realizar la transformación

• Lambdas, procesos simples

• EC2 , con procesos en Java y Python, lo mas

habitual.

• EMR para grandes volúmenes de información

• Glue (a futuro)SERVERLESS

Viesgo identifica tres niveles de usuarios informacionales con diferentes capacidades y diferentes necesidades que requerirán modelos de autoservicio y gobierno diferente.

USUARIOS DE NEGOCIO

POWER USERS

DATA SCIENTIST

Su necesidad está basada en el consumo de información directo a través de herramientas de visualización (Qisght & QlikView & Qlik Sense & Power Bi) .

Evolución deseada - Se pretende incrementar el autoconsumo de estos usuarios y dotarlos de la capacidad de generar informes AD-HOC sin necesidad de colaboración

de las áreas de IT.

Son usuarios con conocimientos técnicos de gestión de información (SQL), tienen la necesidad generar nuevos KPIs y de hacer segmentaciones avanzadas. Sus análisis buscan dar explicación a la realidad de la empresa.

Evolución deseada - Se pretende dotar a estos usuarios de cierta libertad para la realización de análisis ad-hoc con acceso directo a un SET de información mas amplio que los usuarios de negocio y con un espacio de desarrollo propio para realizar análisis . Además deben disponer de un procedimiento estándar para industrializar e incorporar nuevos datos al catalogo de datos oficial.

Son los mas avanzado tecnológicamente y los que realizan os análisis mas sofisticados, suelen realizar análisis de tipo predictivo o segmentación en base nuevas

dimensiones.

Evolución deseada - Se pretende dotar a estos usuarios de capacidades tecnológicas de Advanced Analytics y espacios de desarrollo para realizar sus análisis. Tienen

acceso a un conjunto de información mas amplio que los Power Users y pueden incluir datos externos en sus análisis. El resultado de sus análisis puede ser

industrializado.

Flujo de datos analítico Visualiza y consume……..que tipos usuarios no encontramos y que nos demandan

Flujo de datos analítico: Visualiza y consume……..múltiples tipos de usuarios , múltiples tipos de herramientas, múltiples necesidades

USUARIOS DE NEGOCIO

POWER USERS

DATA SCIENTIST

+

-

Gold y Silver – Gobierno del dato……..como permitimos no comprometer la agilidad con un proceso gobierno del datos riguroso y formal

Nuestro S3 como “source of truth”

Silver

Lugar especifico para proyectos o usuarios donde se almacena información especifica del proyecto o del usuario. Son sanbboxesdonde se puede tener información no gobernada por la organización.

Golden

Lugar donde se establecen los maestros corporativos y donde se ha definido una política de gobierno del dato. Hay un responsable , se tiene definido su linaje..etc etc.

Monitorizar este proceso……..como monitorizar

SERVERLESS


Additional Resources - Whitepaper

Serverless Architectures with AWS Lambda

November 2017

bit.ly/ServerlessLambda

Optimizing Enterprise Economics with Serverless

Architectures

October 2017

bit.ly/ServerlessEconomics

Serverless Applications Lens

November 2017

bit.ly/ServerlessLens

Serverless Streaming Architectures and Best

Practices

June 2018

bit.ly/StreamingServerless


@awscloud_es

Página de eventos y webinars AWS Iberia: https://aws.amazon.com/es/about-aws/events/eventos-es

https://aws.amazon.com/es/about-aws/events/eventos-es/

@awscloud es designing next-gen apps: serverless€¦ · building blocks for serverless...

Documents