containers through the ages - syseleven
TRANSCRIPT
Container through the ages
Christoph Glaubitz Cloud Architect at SysEleven GmbH ContainerDays Hamburg, 28.06.2016
WHAT COULD YOU EXPECT FROM THIS TALK?
A little bit of history on containersHow we used containers in the pastWhat we did to manage our infrastructure
Simplified: Each container gets its own runtimeenvironment, but all share the same kernel.
With the advantage of just running one kernel.
But the disadvantage of just running one kernel.
Same kernel version for all the containers.
It is not possible to load different modules for containers.
OS type in containers must be the same like the host.
(Which is not true any more!)
The most simple way of containering is a traditionalchroot environment. Introduced in 1979.
Which isolates runtime environments, but does not isolateprocess space and devices, and does nothing with
resource control.
In 2000, FreeBSD jails were introduced. chroot wasextended by further isolation mechanisms, like processnamespaces. Processes in one jail can only see processes
in the same jail.
In this time, Virtuozzo started with containers on Linux.
Without a large popularity, similar concepts landed inmainline over the years
Namespaces to limit things, processes can see
cgroupsIPC (2.6.30)Network (2.6.24)Mount (2.4.19)PIDUserUTS
cgroups to limit resources a process can use
cpumemorydevicesblkio[…]
My first contact with containers was around 2007
https://en.wikipedia.org/wiki/Star_Trek:_First_Contact#/media/File:Star_Trek_08-poster.png
When I worked in the Tivoli Storage Manager Team atIBM, where we had to support AIX WPARs and Solaris
Zones.
The ideas are still the same, also in the times ofApplication Containers.
Using the same kernel.Isolate environments as strong as possible by using thementioned namespaces and cgroups.
WHY DID CONTAINERS GET THIS TRACTION INTHE PAST FEW YEARS?
I think, it the easy way to create and deploy images.
https://www.flickr.com/photos/smemon/
We can run the image (almost) everywhere!
… in a chroot without isolation and resource control usingrkt fly by CoreOS
… in a small KVM (with an injected kernel), again using rkt,thanks to CoreOS and Intel
… on FreeBSD using JetPack
… even on Windows!
And all this without any modification to the image!
It changed the way we think about software.
One application in one container.
Only one process where possible.
Applications being stateless…
… and more fault tolerant.
Smaller chunks of source code doing one thing…
… which could be deployed independent and often.
BUT THERE WILL BE DRAGONS!
Unfortunately it is also very easy to build crappy images!!
https://www.flickr.com/photos/57402879@N00/
In my experience, deployments fail more likely because ofmisconfigured next-hops (like DB), instead of crappy
software.
Plugging services together is even more difficult in afloating environment.
More management software is needed!
I strongly encourage you to read on this.
https://charity.wtf/2016/05/31/wtf-is-operations-serverless/
https://charity.wtf/2016/05/31/operational-best-practices-serverless/
"Microservices: because solving business problems is hardbut building loosely coupled fault-tolerant distributed
systems is easy."
https://twitter.com/neil_conway/status/743086761493008384
SO… WHAT ARE YOU GUYS AT SYSELEVENDOING?
RUNNING TWO TYPES OF INFRASTRUCTURE!
A NEW ONE, BASED ON OPENSTACK…
… AN OLD ONE, BASED ON PARALLELSVIRTUOZZO…
… but most important: Provide, maintain and monitor theinfrastructure into which our customers can deploy their
code into.
ON TWO LEVELS!
1 . The hardware and hypervisor "layer".
2 . The running Virtual Environments/containers, with allthe services in it.
LET'S STEP INTO THE VIRTUOZZO PLATFORM
In the past, Virtuozzo was basically
image formatlocal storagenetwork isolationLinux based container runtime(but maintained outside of mainline)
… treating containers like usual virtual machines
with a complete Linux running insidestarting with the init systemsure, except of the kernel
target distribution has to be supported by virtuozzoto set up the network, hostname, etcthis is extensible by shell scripts
But with a huge performance benefit over VMs
https://www.flickr.com/photos/17612257@N00/
SOME MORE HISTORY
From the early days, we have decent trending and alertingfor all the services running in the containers.
We even built a web frontend, which gathered data frommultiple nagios instances to give us a single overview.
… and an AdminTV with relevant graphs
We run a pool of hardware nodes…
… on which we schedule the containers of all customers.
https://www.flickr.com/photos/prinsotel/
In the beginning we track the containers and hardwarenodes in some kind of sheet.
Creating a new container was like…
Having a look into the sheet, and select a host that doesnot run the same kind of instance of this customer
Having a look at the metrics to verify the host still hasresources
ssh into the host and create the container, using thevirtuozzo tools
Assigning an ip and a hostname like customer.project.app1
Enter the container, and install stuff by hand
Register the new container with its services to themonitoring by hand
… we developed some tools to automate most of this stuff
… with a recommend system to return the best Hardwarefor the requested container.
https://www.flickr.com/photos/jm3/
Hostnames are partly calculated. customer.project is pre-filled, but the name app2 has to be selected by hand.
We came up with a tagging system to be a bit moreflexible.
But we still totally lack a customer facing API.
Everything works via phone and ticketing system.
https://www.flickr.com/photos/nekudo/
Over time, we built billing, a local DNS-API, SSL-Cert"shop", some own metering-endpoints, Backups…
At some point, Configuration Management became cool
… and we added a Puppet Master to our infrastructure,and registered all the containers to it. Based on this, weautomated much stuff, like provisioning of basic stuff tothe container and registering services to the monitoring.
We wrote much glue-code
https://www.flickr.com/photos/samcatchesides/
But we still treat the containers like computers.
THESE DAYS WE ARE RUNNING
HOW DOES A TRADITIONAL SETUP LOOK
MySQL master slave setupAdmin servera bunch of App servermaybe some kind of search engine
We start the containers, using our internal toolchain.
This also triggers puppet to install required software inthe container, setting up configuration and register them
to our monitoring.
We hand over exactly this container to the customer.
https://www.flickr.com/photos/lindzgraham/
Customers deploy their software to the admin server.From there it will be rsynced to the app server.
https://www.flickr.com/photos/auxesis/
Same thing for dev or test setups.
https://www.flickr.com/photos/christianjann/
All this worked very well for a long period of time!
But is very inflexible!
Installation of containers is strongly coupled to ourinfrastructure.
So changes to the infra hurt.
We have startup dependencies rather than builddependencies.
Don't get me wrong!
You can build such problems in the shiny new containerworld as well!
Often a new container is just a clone of an old one!
https://www.flickr.com/photos/arenamontanus/
With things like hostname and monitoring stuff changed
There is
in it.
https://www.flickr.com/photos/trevorandmarjee/
So we keep the containers alive, no matter what happens.In the worst case, we clone one back from the nightly
backup.
https://www.flickr.com/photos/vagawi/
Updates have to go to all containers…
https://www.flickr.com/photos/bovinity/
… rather than building one new image and replace oldcontainers.
https://www.flickr.com/photos/nicowa/
Because customers software is not packaged in any way…
… there is no overview of the versions of the deployedsoftware.
Rollbacks to a defined set of installed software anddeployed software are nearly impossible.
https://www.flickr.com/photos/thejesse/
Anyway.
This again worked fine for a very long time…
But these days we need a public facing API
We didn't have the man power to develop our own…
… and would have to replace too much of the inner core.
And we did not want to build another proprietary
again!
https://www.flickr.com/photos/simuh/
So we came up with OpenStack, and created…
We run managed setups on SysEleven Stack.
Some customers choose to run production fully managed,and dev on self managed instances.
The core concept of running the managed setups is prettymuch the same as before.
https://www.flickr.com/photos/yoroy/
But with lessons learned from the old platform!
https://www.flickr.com/photos/pictoquotes/
The setups are flexible, and we can also run them on AWS.
It based on ansible, which creates VMs using the specificAPI, and runs the deployment in it.
We enable customers to be able to do relevant tasks ontheir own, and get the full power out of the Cloud.
Despite of this…
…in the past half year one question raised…
What about
?
Basically you can run every OS you like to. Yet Windows.
SO. YES!
Just install docker into the VM.
Sure. The containers run in VMs. But there is IMHO nocontainer vs. VM, while VMs are one possible
infrastructure for containers.
Are there future plans to provide managed containerservices?
OF COURSE
In a first step, we will help you to get the orchestrationlayer up and running.
We help you with setting up monitoring.
We yet know what cantainers are. We know the tools andhow to debug in the container world.
We will help you to get your deployment chains to work.
But that is not "managed conatainer services" at all!
We think about providing managed orchestration layers.
Like
… or Docker Swarm
We would manage and monitor all the services of theconductor.
We may manage additional services like a gitlab, jenkinsor a private image registry.
The customer still is responsible all for the containerworkloads.
Maybe in a far
https://www.flickr.com/photos/79909830@N04/
We could provided managed services running incontainers.
But there are many things to think about!
https://www.flickr.com/photos/dharmabum1964/
SOME RESOURCES
https://charity.wtf/2016/05/31/wtf-is-operations-serverless/
https://charity.wtf/2016/05/31/operational-best-practices-serverless/
https://twitter.com/neil_conway/status/743086761493008384
http://kubernetes.io/
https://docs.docker.com/swarm/
https://github.com/chrigl/heat-examples
https://chrigl.de/slides/sysconf15-docker/#/ecosystem
https://chrigl.de/slides/sysconf15-docker/#/resources
HAPPY
