docker {at,with} signalfx
TRANSCRIPT
![Page 1: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/1.jpg)
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
DOCKER {AT,WITH} SIGNALFX
![Page 2: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/2.jpg)
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
Me?
github.com/mpetazzoni
@mpetazzoni
![Page 3: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/3.jpg)
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
What is SignalFx?
![Page 4: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/4.jpg)
The pitch
• Real-time monitoring system for modern applications
• User-defined, streaming and interactive analytics
• Alerting and anomaly detection
• Easy to deploy, to use and to integrate
• Scales; from small deployments to large, complex infrastructures
![Page 5: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/5.jpg)
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
Docker at SignalFx
![Page 6: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/6.jpg)
Docker at SignalFx
• Started using Docker in November 2013
• Running Docker in prod ever since we’ve had a “prod”
• Back when Docker’s README said “DO NOT RUN IN PRODUCTION”
![Page 7: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/7.jpg)
Objectives
• Separate infrastructure concerns
• Stronger independence from hosting provider
• Development lifecycle
• Application packaging and delivery
• Higher-level orchestration
![Page 8: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/8.jpg)
Infrastructure vs application
• Separation of concerns
• Infrastructure provides appropriately spec-ed hosts and network
• Abstracts from cloud provider
• Currently on AWS
• Amazon Linux AMIs + Salt to setup a few things, up to Docker
![Page 9: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/9.jpg)
Infrastructure vs application (cont.)
• “Application” stack takes it up from there
• Everything is containerized, including 3rd party components
• No dependence on Amazon specifics
• We can spin up the stack anywhere (laptop, GCE, DO, …)
![Page 10: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/10.jpg)
SDLC
• Run locally, in tests and in prod environments
• Same bits, same tools, same arguments, etc.
• Crucial for software validation
• Maven, Jenkins, Quay.io, MaestroNG
• maestro start
![Page 11: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/11.jpg)
SDLC (cont.)
docker-maven-plugin
![Page 12: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/12.jpg)
SDLC (cont.)
• Continuous build of the release branch
• Builds and pushes :latest tagged Docker images
• Promotion track through :lab, :prod and :mon
• One-click promote + upgrade, orchestrated by MaestroNG
• Automated integration tests environment
![Page 13: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/13.jpg)
Application packaging and delivery
• All components of the stack are packaged as Docker images
• 3rd-party component packaging was figured out once (Dockerfile)
• SignalFx components all have their Dockerfile too
• Built by docker-maven-plugin
• All components have a run.py init script
![Page 14: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/14.jpg)
Discovery
• ZooKeeper / Curator service discovery
• We built announcing wrappers for Cassandra and ElasticSearch
• Only need to know “realm” name and zkConnectString
• Everything else is discovered from ZooKeeper
• Need to make sure to announce host IP address/port • Maestro gives us that via environment
![Page 15: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/15.jpg)
Orchestration
• We built MaestroNG
• Orchestrator of multi-host Docker environments
• github.com/signalfuse/maestro-ng
• Jinja2/Yaml environment file
• Defines what runs where, and how
• Knows and respects service dependencies
![Page 16: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/16.jpg)
MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
Monitoring containers
![Page 17: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/17.jpg)
The basics
• Docker’s /stats API provides the bare minimum • CPU
• Memory
• Network I/O
• Disk I/O
• Pitfalls; GET /stats is blocking, up to a second
• Use CollectD and docker-collectd-plugin • github.com/signalfx/docker-collectd-plugin
![Page 18: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/18.jpg)
The basics (cont.)
• Data returned by /stats API needs “math” to yield usable numbers • See Docker client code
• Better: use SignalFx! • Docker Dashboard
• Collect host metrics too if you can (with CollectD)
• System/container-level metrics are very limited
• We need to go deeper
![Page 19: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/19.jpg)
Application monitoring
• Usually needs more specific visibility
• Instrument first, ask questions later
• Metrics are cheap, both in CPU and $
• For 3rd-party apps, use CollectD and its plugins • We’re doing a lot of work to improve/provide CollectD plugins for common FOSS
• And provide curated dashboards for each of them
![Page 20: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/20.jpg)
Application monitoring (cont.)
• For 1st-party software, depends on language platform
• For VM-based services (Java, Scala, Go, …), get key metrics • Heap
• # threads
• Time spent in GC and # collections
• Use metric libraries and instrument everything
• Bring it all together in dashboards and anomaly detectors
![Page 21: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/21.jpg)
Docker with SignalFx
• Objective: time-to-data as short as possible
• Easiest route: use our packaged CollectD • Good default set of host metrics
• Install/enable plugins as needed; docker-collectd-plugin
• SignalFx automatically imports curated dashboards
• That’s it!
![Page 22: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/22.jpg)
More monitoring with SignalFx
• Use our client libraries or API to send your monitoring data • Or tee off your existing Graphite, StatsD, … pipeline towards SignalFx
• Leverage the power of the real-time, streaming analytics
• Correlate system, container and application metrics
• Build better dashboards and alerts from higher-level signals
![Page 23: Docker {at,with} SignalFx](https://reader031.vdocuments.mx/reader031/viewer/2022030216/588901dc1a28abcf5f8b6599/html5/thumbnails/23.jpg)
Thanks!
github.com/signalfuse/maestro-ng
github.com/signalfx/docker-collectd-plugin
signalfx.com/free-trial.html
@signalfx — signalfx.com/careers.html
@mpetazzoni — github.com/mpetazzoni