Download - Logging & Docker - Season 2
![Page 1: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/1.jpg)
Sumo Logic Confidential
Logging & Docker
Christian Beedgen, CTO & Co-Founder, Sumo Logic
Seattle Docker Meetup, October 13, 2015
![Page 2: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/2.jpg)
Sumo Logic Confidential
$ whoami• Co-Founder & CTO, Sumo Logic
Cloud-based Machine Data Analytics ServiceApplications, Operations, Security
• Chief Architect, ArcSightMajor SIEM player in the enterprise spaceLog Management for security and compliance
![Page 3: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/3.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 4: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/4.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 5: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/5.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 6: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/6.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 7: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/7.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 8: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/8.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 9: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/9.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 10: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/10.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 11: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/11.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 12: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/12.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 13: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/13.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 14: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/14.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 15: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/15.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 16: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/16.jpg)
Sumo Logic Confidential
December 2014, New York City
![Page 17: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/17.jpg)
Sumo Logic Confidential
December 2014, New York Cityhttp://www.slideshare.net/
raychaser/6-million-ways-to-log-in-docker-nyc-docker-meetup-12172014
![Page 18: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/18.jpg)
Sumo Logic Confidential
Season 2Where Are We In Late 2015?
![Page 19: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/19.jpg)
Sumo Logic Confidential
Basics• Logging in Docker as per 12factor.net
![Page 20: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/20.jpg)
Sumo Logic Confidential
Basics• Logging in Docker as per 12factor.net
![Page 21: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/21.jpg)
Sumo Logic Confidential
Basics• Logging in Docker as per 12factor.net
• Also, one process per container, plz!
![Page 22: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/22.jpg)
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
![Page 23: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/23.jpg)
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
![Page 24: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/24.jpg)
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
![Page 25: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/25.jpg)
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
![Page 26: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/26.jpg)
Sumo Logic Confidential
Pre-Docker 1.6
• Early hardcore crowd would just collect /var/lib/docker/containers/**
• And then of course there’s the UX: docker logs• docker logs is using a daemon API for getting the logs• This leads to logspout – attach to API, forward to Syslog• https://github.com/gliderlabs/logspout
![Page 27: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/27.jpg)
Sumo Logic Confidential
Docker 1.6 Introduced Log Drivers
• Hallelujah• Initially supports json-file, syslog, null• json-file – default, this is the old mechanism
– Continues to this day to be required for API access and docker logs• docker run -–log-driver syslog …
– Sends to local Syslog, no more writing to disk• docker run –-log-driver null
– STFU, basically
![Page 28: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/28.jpg)
Sumo Logic Confidential
Docker 1.7 Introduces --log-opt
• Now we can pass parameters to the log drivers!• docker run \
--log-driver syslog \
--log-opt syslog-address=(udp|tcp)://… \
--log-opt syslog-facility=(kern|daemon|user|local0|…) \
--log-opt syslog-tag=“myapp”
• Forward directly to local Syslog aggregator, or to a cloud-based logging service
• Docker 1.7 also added support to log to journald
![Page 29: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/29.jpg)
Sumo Logic Confidential
Docker 1.8, 1.9 - Even More Log Drivers
• Fluentd
• GELF
• AWS
![Page 30: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/30.jpg)
Sumo Logic Confidential
Also in Docker 1.8 – Options For json-file
• json-file still the default, still required for docker logs and /logs API• Long standing problem – will eventually fill up your disk• Folks have been using logrotate hacks…• Now, json-file log driver can be configured:
• Basically, keep up to max-file files, roll current at max-size
![Page 31: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/31.jpg)
Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
![Page 32: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/32.jpg)
Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
![Page 33: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/33.jpg)
Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
![Page 34: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/34.jpg)
Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
![Page 35: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/35.jpg)
Sumo Logic Confidential
What Is Sumo Working On?
• We have containerized our collectors– https://github.com/SumoLogic/sumologic-collector-docker– docker run -d -p 514:514 -p 514:514/udp \
--name="sumo-logic-collector" \
sumologic/collector:latest-syslog \
[Access ID] [Access key] – https://www.sumologic.com/2015/09/09/update-on-logging-with-docker/
![Page 36: Logging & Docker - Season 2](https://reader036.vdocuments.mx/reader036/viewer/2022062522/587b44421a28ab9c0e8b65b9/html5/thumbnails/36.jpg)
Sumo Logic Confidential
What Is Sumo Working On?
• We are working towards our vision of Comprehensive Monitoring– https://www.sumologic.com/2015/06/16/comprehensive-monitoring-for-docker-more-than
-just-logs/
• We have released an initial App for Docker at DockerCon 2015