experiences with serverlessbig data · be serverless and serve data amazon s3 aws lambda aws lambda...

Munich, 17.10.16Markus Schmidberger, Head of Data Service

ExperienceswithServerless BigDataAWS Meetup – Munich 2016

2glomex – A company of ProSiebenSat.1 Media SE

Key Components of our Data Service

ContentDiscoveryFindthemostrelevantcontentforourcustomersandtheirusers.

Real-TimeMonitoringEnableourdevelopmentteamstoserveourcontenttoourusersinthebestqualitypossible.

AnalyticsProvideourteamsaccesstothedatatoenabledata-drivendevelopmentofnewfeaturesandproducts.


Micro-Service Architecture

INGEST STOREPROCESS &

ANALYSEVISUALIZE &

SERVE

AdProxy Log Import Service

Player Feedback Import Service

Data PlatformAccess

Data ScienceAnalytics Service

TechnicalMonitoring

Service

Dev / Ops Analytics Service

Content Discovery Service

KPI & Analytics Service

MetadataService

ContentImport Service

Data Platform Monitoring Service

Data QualityService

Data Management

Service

Data Layer

Data API

Data Lake

External Data Import Service

Portal

CDN files

data stream

data stream

Team

VAS Log Import Service

data stream

other modules

Real-Time Dashboards

ContentAPI

Data Platform - MicroService Layout

CDN Log Import Service

Data Science UI


Lambda Architecture

BatchProcessing• KPIsforMES• MESBilling

DataScience• CDNBills• DataInsights

Real-time• Real-timeplayer

monitoring• Internaldashboard


Lambda Architecture

Graphic provided by http://lambda-architecture.net

≠AWSLambda


AWSLambda

Notification

Amazon S3 AWS Lambda processes the object

Amazon S3

New object uploaded

Amazon DynamoDB


AWSLambdaExecution


Be serverless for ETL


Be serverless and serve data

AWSLambda AWSLambda AmazonAPIGatewayAmazonS3


Be serverless for Recommendations

Recommendation Pipeline

Publisher’sURL RecommenderSystem

SearchDownloadPagecontent

ContentPlaylist


• Read data from Kinesis Firehose / S3

• Server downtime / scheduler

• Load to ElasticSearch

• Clean ElasticSearch and Redshift

• Advanced Redshift monitoring

• EBS Snapshots

Be serverless everywhere


Agile DevOps

Continous Delivery of Micro-services – „Automate all the things“


Agile Cloud Deployment

Glomex Cloud Deployment Tools

KumoAWSCloudFormation

RamudaAWSLambda

YugenAWSAPIGateway

TenkaiAWSCodeDeploy

• IncooperationwithOperationsTeam

• Usedbyotherteams

• Simplifyclouddeploymentsdrastically

• Slack- andmonitoringintegration


Agile DevOps

Cross-functional team responsible to push components to production themselves


AWS Lambda Limits

5 min512 MB

AWS Lambda Timeout

AWS Lambda temp disk

• Howtoprocess800MBgziped logfile?

• Howtosplitcompressedgzip files?

• SplitterusingAmazonSQSandAmazonEC2SpotInstances


• Lambda function deployment package size (.zip/.jar file)• 50 MB

• Total size of all the deployment packages that can be uploaded per region• 75 GB

• CreateLogGroup• 500 log groups / account / region

• Lambda functions have to be wired• Be aware of retries

• 3 not configurable• Traceback and error output available via CloudWatch Logs

• https://github.com/jorgebastida/awslogs• No local development environment

More AWS Lambda limits


More Facts

20 GB5 Billion

Per day click-stream data IN (player, vas, adproxy)

Click-stream records processed per day

~100 ms Data freshness to S3

25 GB300 Million

Per day as zipped CDN log-files

CDN record processed per day

< 1 min Data freshness to API


More Facts

600 rec/sec

1 $ / hour

Processing time

Cost for 25 GB/dayCDN processing

6 Parallel AWS Lambda functions

2.3 min Average run-time of AWS Lambda AWS Lambda duration

Redshift CPU

~ 89% - 97.8% Accuracy


Key Takeaways

LambdaArchitecture

Enrichyourtraditional,batch-drivenBI-workflowwithreal-timeanalytics

UseLambda-Architectureasaguidingprincipleandadaptittoyourneeds


Analyze

TakeActionsAutomate

1

23

Key Takeaways


Key Takeaways

AWSmanagedservicesprovideanrobustwaytoruncomplexbigdatainfrastructures

Followbest-practicesprovidedbyAWSandthecommunity

Focusonfeaturesdevelopmentandrobustpipelinesnotoninfrastructuremanagement

Munich, 17.10.16Markus Schmidberger, Head of Data Service@cloudHPC,[email protected]

Wearehiring…

• DataEngineers

• ProductOwner

• BigDataProductLineManager

experiences with serverlessbig data · be serverless and serve data amazon s3 aws lambda aws lambda...

Documents