Zenoss Control Center

Open Source Application Management & Orchestration System for Docker

controlcenter.io

Overview

• History• Brief introduction to Docker • Why Docker? • What is Control Center? • Demo• Questions

History

• Zenoss founded in 2005 as Open Source Enterprise IT Management Solution

• Docker was chosen to support the scale requirements for monitoring the modern datacenter

• Need was created for managing Zenoss within Docker

• Birth of Control Center • Zenoss Control Center Open Sourced in August

2014

Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application.

What is Docker? docker.io

http://slides.com/stevenborrelli/docker

● Based on Linux containers● Minimal overhead (cpu/io/network)● Can run on any Linux system that supports LXC, nsinit● Docker container contains everything it needs to run● A docker container should be able to run anywhere that

Docker can run.

What is Docker?

http://slides.com/stevenborrelli/docker

Why Docker?

• Scalability– Ability to spin up new daemons on the fly

• Easy Installs – Just pull image and deploy– Dependencies exist in the container

• Reliable History Tracking– Improves upgrades– Easy to roll back

What Docker doesn’t provide (yet)● service discovery

● a good log interface

● cross host volumes

● cross host scheduling

… and anything else you may find in a cluster manager

Where Docker Ends, Control Center Begins

Physical/Virtual Server

Host OS (UOS)

Container Manager

Control Center

App Containers

CC

Control Center (CC) is an application written in Go! (aka golang).

CC takes care of all interactions with Docker and the host system.

You install CC on every computer that you intend to run Zenoss on.

CC is open source software (Apache 2 license).

Terms● serviced: Control Center command-line client

● Service: An entity that manages a container and/or other services.

● Service Definition: Describes how a service is run and configured

● Service Template: A document of service definitions relating to a platform.

We tell Control Center how to execute the application inside of a container image via a Service Template.

The same container may act completely differently depending on the service definition used to start it.

CONTROL CENTER - SYSTEMS

The Control Center has 3 important systems that divide the work of orchestrating and monitoring of services.Master - This subsystem houses several subservices and is generally responsible of the overall orchestration of the system. There will only be one Control Center Master.

Node Controller - This controller is responsible for all docker interaction, starting/stopping services, mounting of service resources and metric collection. There is one of these per host.

Container Controller - This program is the first program that gets launched inside a container. It is responsible for starting the actual service, providing network proxies, forwarding logs/events, and forwarding service specific stats.

CONTROL CENTER - SYSTEM OVERVIEW

MasterLogstash

Zookeeper

Metric Svc

Docker Registry

UI/Rest API

RPC APIDocker

Node Controller

RPC API

Container

controller

Your Service

logstash-forwarder

Container

controller

Your Service

logstash-forwarder

Container

controller

logstash-forwarder

Your ServiceTCP Mux

Shared FS Host Boundary

Node Controller TCP MuxRPC

CONTROL CENTER MASTER

Elasticsearch Database for control center

Zookeeper Coordination service

Logstash Log consolidation Uses its own copy of Elasticsearch

Metrics -- CC centric metric system OpenTSDB Standalone Hbase Consumer Query service

Celery Schedule tasks

Registry docker-registry

Shared Storage (NFS)

Web server for CC UI

Rest API / RPC API

CC NODE CONTROLLER

● 1 Node Controller per host

● Configured how to talk to master

● Watches the master’s zookeeper for services that has been scheduled to start on its hosts

● Pulls down images from Master’s Docker Registry

● Starts/Stops/Manages services as needed

● Mux distributes all network traffic to this box to the appropriate containers

CONTAINER CONTROLLER

● Injected into each container

● Serviced binary is bind mounted into every container that it runs automatically

● The Serviced container controller is what runs and then executes the actual service command

● Imports endpoints from other services

● transparently proxies connections to remote endpoints and makes them appear local

● Bind mounts Logstash forwarder

CC - CORE CONCEPTS

● IP Addresses● Hosts● Storage Systems (future)

Pool - a collection of compute, network and storage resources.

CC - CORE CONCEPTS

Host - a compute node (physical or virtual) that is available to the control center.

○ Hostname○ Pool ID○ Kernel Version○ OS Distribution○ Docker Version○ ServiceD Version○ Memory Size○ CPU Count○ Interface (NICs) Names & Addresses○ Monitoring Profile

https://github.com/zenoss/serviced/blob/develop/domain/host/host.go#L19

CC - CORE CONCEPTS

Service Template - A hierarchical description of a services that deliver an Application

Resource Manager

Zproxy (Nginx)

Zope

MariaDB

ZEP

RabbitMQ

MariaDB 2

Query OpenTSDB HBase

Zookeeper

Not Limited to Tree Dependencies

CC - CORE CONCEPTS

● ImageID - what container image does this service use● Startup - what command must run to start the service● Config Files - what configuration files to inject into the container

before startup● Log Files - what log files the service writes to inside the container and

what is the format of those log files● Memory Limit - what is the maximum allowable amount of memory

the service can use● Volumes - what persistent disk storage is needed by the service and

where that storage should be mounted● Endpoints - what network endpoints does this service depend on or

does this service expose● Health Checks - what scripts can be run to determine the health of a

service● Metrics - what datapoints the service exposes● Thresholds - what thresholds can be applied to the exposed metrics● Graphs - what graphs are relevant to the service

Service - Is a description of an individual process.

CC - CORE CONCEPTS

Schema - the relationship between the various domain objects in the control center

Pool [default]

Host [app0.example.com]Host [app0.example.com]

Host [app0.example.com]

Control CenterService Template

Service AService A

Service A

ServiceA @T0

CC - NETWORK CONNECTIONS

Your Service

Imagine your service needs to talk to Redis.

localhost interface (lo)127.0.0.1

Container Controller

It thinks it can access redis via 127.0.0.1:6379The container controller actually accepts the connection

Zookeeper

The controller finds where redis is actually running via ZK.

Mux

The controller makes an encrypted connection to the mux.

Redis Container

The mux contacts the exposed port of the redis container.

MariaDB Sample Service Template{

"Command": "/usr/bin/mysqld_safe","Endpoints": [

{ "Application": "mysql", "PortNumber": 3306, "Protocol": "tcp", "Purpose": "export" }

],"ImageId": "zenoss/zenoss5x","Instances": {

"min": 1},"Launch": "auto","LogConfigs": [

{ "path": "/var/log/mysqld.log", "type": "mysql" }

],"Name": "MySQL","Snapshot": {

"Pause": "PAUSE_CHECK_TIMEOUT=60 LOCK_HOLD_DURATION=600 $ZENHOME/bin/quiesce-mysql.sh pause", "Resume": "$ZENHOME/bin/quiesce-mysql.sh resume"

},"Tags": [

"daemon"]

}

Persistent VolumesFor those services that have to maintain state between

execution, the Control Center will mount a folder from the Shared File System into the container.

The SFS is used so that even if the service starts up on a different machine it can still access the same files.

Operating System

Control Center

DockerSFS

S

ervi

ce 1

S

ervi

ce 2

Control Center

S

ervi

ce 5

Operating System

DockerSFS

S

ervi

ce 3

S

ervi

ce 4

S

ervi

ce 6

Install Control Center on Ubuntu 14.04

# install docker repowget -O - http://get.docker.io | sh

# install zenoss reposudo apt-key adv --keyserver keys.gnupg.net --recv-keys AA5A1AD7sudo sh -c 'echo "deb [ arch=amd64 ] http://get.zenoss.io/apt/ubuntu trusty universe" \ > /etc/apt/sources.list.d/zenoss.list’

# update repo databasesudo apt-get update

# install Control Centersudo apt-get -y install serviced

# then start servicessudo start serviced

Demo Time

Questions?