Centralized logging solution on Mesos using ELK stack

Imran ShaikhLead Systems Engineer

YellowPages

Issues?• No consolidated view

– The problem gets worse if you are running multiple instances of an app.• How do you debug the issue?

– Which container has the problem?• Lets say you are able to identify the container and there is nothing on

STDOUT/STDERR:– how would you know its an issue with system, docker or mesos?– How do you stitch all this information together?

• Lets say you are able to stitch it together:– how do you know which mesos task and marathon app this container

belongs to?• There are multiple docker logging solutions, but none of the truly fits

Mesos landscape.

Mesos SlaveDocker Container 1 Docker Container nDocker Container 2

YP Logspout Container

Syslog

Logstash forwarder Container

Logstash Cluster

ElasticSearch Cluster

Kibana Container 1 Kibana Container 2 Kibana Container 3

Logstash 1 Logstash 2 Logstash 3 Logstash n

ES 1 ES 2 ES 3 ES n

Mesos DNS

System logs

Mesos logs

Docker logs

What are we doing at YP Engineering?

• Writing bunch of Mesos supported technologies:– Logging– Dynamically provisioning Application Secrets to

containers– Persistent storage– Distributed Monitoring & Alerting– Metrics– Self-serve– Etc.

• Open source contribution:www.github.com/yp-engineering

• You can reach us at:IRC: chat.freenode.net (#yp-engineering)Twitter: @imranshaikhEmail: ishaikh@yp.com, ishaikh@gmail.com

• Any questions?We have a big contingent of a team here.