centralised logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/paul stack...

20
Centralised Logging… without the blood, sweat and tears Paul Stack http://twitter.com/stack72 mail: [email protected]

Upload: truongliem

Post on 17-Mar-2018

230 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

Centralised Logging…

without the blood, sweat and tears

Paul Stackhttp://twitter.com/stack72mail: [email protected]

Page 2: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

Paul Stack @stack72

Page 3: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

Why Centralised logging?

Page 4: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another
Page 5: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

What happens if you lose a server and the logs were only on that server?

Page 6: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another
Page 7: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

Simple Logging Architecture

Page 8: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another
Page 9: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another
Page 10: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

Streaming Architecture

Page 11: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another
Page 12: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

There has to be another way…

Page 13: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another
Page 14: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

AWS ElasticSearch

AWS Lambda

AWS Kinesis Stream

Application Servers

Page 15: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

Code Time…

Page 16: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

How do I send logs to Kinesis?

Page 17: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

Costs of running this…

Page 18: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

• Kinesis Streams - $0.015 for every million PUT • Kinesis Streams - Extended Data $0.02 • Lambda - $0.22 for every million queries • ElasticSearch - $0.09 per hour per node • ElasticSearch - $0.135 per GB / per month

$ 144.87 per month

Page 19: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

Questions?

Page 20: Centralised Logging… - schd.wsschd.ws/hosted_files/buildstuff16ukraine/da/Paul Stack @stack72... · Simple Logging Architecture. Streaming Architecture. There has to be another

Paul Stack @stack72