99cloud docker training module 2
TRANSCRIPT
COURSE INTRODUCTION
DAY 01• Docker Introduction• Installation of Docker• Working with Containers• Working with Images• Building image with
Dockerfiles• OpenStack Integration• OpenStack Introduction• Nova Driver (Docker as
Compute Service)• Glance as Docker image
registry
DAY 02• Docker Cluster• Kubernetes Introduction• Docker Networking• Docker Hub• Docker Cases• Docker Source Code• Development Platform
With Docker• Hadoop on Docker
Docker Cluster
Docker Basic
Now that we have containers
• Isolation: Keep jobs from interfering with each other• Scheduling: Where should my job be run?• Lifecycle: Keep my job running• Discovery: Where is my job now?• Constituency: Who is part of my job?• Scale-up: Making my jobs bigger or smaller• Auth{n,z}: Who can do things to my job?• Monitoring: What happening with my job?• Health: How is my job feeling?
Proposed implementation strategy
Cluster
• Management Layer• Zookeeper• etcd
• Storage• Btrfs• Ceph• ZFS
• Networking• Open vSwitch
• Live Migration
Kubernetes
Manage a cluster of Linux containers as a single system to accelerate Dev and simplify Ops
What is Kubernetes
Kubernetes is an open source orchestration system for Docker containers. It handles scheduling onto nodes in a compute cluster and actively manages workloads to ensure that their state matches the users declared intentions. Using the concepts of "labels" and "pods", it groups the containers which make up an application into logical units for easy management and discovery.
Enter Kubernetes
• Container orchestrator• Runs Docker containers• Supports multiple cloud and bare-metal environments• Inspired and informed by Google’s experiences and internal systems• Open Source, written in Go• Manage applications, not machines.
Kubernetes Origins
• Runs hundreds of thousands of jobs• For many thousands of different applications
(YouTube, Search Index, Maps, etc.)…• Across a number of clusters (hundreds)• Borg manages many millions of physical
servers.• In production since 2004
Kubernetes Design principles
• Declarative > imperative: State your desired results, let the system actuate• Control loops: Observe, rectify, repeat• Simple > Complex: Try to do as little as possible.• Modularity: Components, interfaces, & plugins.• Network-centric: IP addresses are cheap• No grouping: Label are the only groups.• Bulk > hand-crafted: cattle vs pets• Open > Closed: Open Source, Standards, REST, JSON, etc.
Concept
• Container: A sealed application package (Docker)• Pod: A small group of tightly couple Containers• Controller: A loop that drives current state towards desired state.• Service: A set of running pods that work together.• Labels: Identifying metadata attached to other objects.• Selector: A query against labels.
Pod
• Small group of containers & volumes Tightly coupled • same node
The atom of cluster scheduling & • placement
Shared namespace • share IP address & localhost Ephemeral • can die and be replaced
Example: data puller & web server
Pod Networking
• Pod IPs are routable• Docker default is private IP • Pods can reach each other without NAT • even across nodes
• No brokering of port numbers • This is a fundamental requirement • several SDN solutions
Pod Networking
• Arbitrary metadata • Attached to any API object • Generally represent identity • Queryable by selectors
• think SQL ‘select ... where ...’ • The only grouping mechanism
• pods under a ReplicationController• pods in a Service • capabilities of a node (constraints)
• Example: “phase: canary”
Replication Controllers
f0180
node1
b3112
node2
c1209
node3
a65321
node4
Replication Controller- Desired = 4- Current = 4
Replication Controllers
f0180
node1
b3112
node2
c1209
node3
a65321
node4
Replication Controller- Desired = 4- Current = 4
Replication Controllers
f0180
node1
b3112
node2
a65321
node4
Replication Controller- Desired = 4- Current = 3
Replication Controllers
f0180
node1
b3112
node2
a65321
node4
Replication Controller- Desired = 4- Current = 4
c1209
Services
• A group of pods that act as one == Service • group == selector
• Defines access policy• only “load balanced” for now • Gets a stable virtual IP and port
• called the service portal • also a DNS name
• VIP is captured by kube-proxy • watches the service constituency • updates when backends change
• Hide complexity - ideal for non-native apps
How to deploy
• Prerequisites• The minion nodes have installed docker 1.2+• All machines can communicate with each other• Dependences: etcd-2.0.0, flannel-0.4.0, k8s-0.15.0• All the remote servers can be ssh logged in
Main Steps
• Make kubernetes, etcd and flanneld
• First clone the kubernetes github repo,• $ git clone https://github.com/GoogleCloudPlatform/kubernetes.git then
$ cd kubernetes/cluster/ubuntu.• Then run $ ./build.sh, this will download all the needed binaries into ./binaries.
Configure Cluster
• export nodes="[email protected] [email protected] [email protected]" • export roles=("ai" "i" "i") • export NUM_MINIONS=${NUM_MINIONS:-3} • export PORTAL_NET=11.1.1.0/24 • export FLANNEL_NET=172.16.0.0/16
• KUBERNETES_PROVIDER=ubuntu ./kube-up.sh
Trouble shooting
• Generally, what this approach did is quite simple:• Download and copy binaries and configuration files to proper dirctories on every node• Configure etcd using IPs based on input from user• Create and start flannel network• So, if you see a problem, check etcd configuration first
Docker Networking
Key Takeaways
• Docker networking is in early stage• Application must choose what networking is right for their needs. It is possible to use
same principles as VMs• Open vSwitch brings powerful networking capabilities• User space vs Kernel space packet processing is an important design choice
High-level conceptsNamespace Containerized0networking0at0the0process0level0managed0at0/proc
Linux0Bridge L2/MAC0learning0switch0built0into0the0Kernel0to0use0for0forwarding
OpenvSwitch Advanced0bridge0 that0is0programmable and0supports0 tunneling
NAT Network0address0translators are0intermediate0entities0that0translate0IP0address0+0Ports0(Types:0SNAT,0DNAT)
iptables Policy0engine0 in0kernel0 that0is0used0for0managing0packet0forwarding,0 firewall,0NAT0features
Unix domain0sockets
File0descriptor0based0communication0 that0is0restricted0to0a0single0host.0Works0 like0a0FIFO0pipe.
UserRspace vs0KernelRspace
Application0domain0 that0regulates0access0to0resources0and0performance0possible.0• Container0applications0 run0in0userRspace• Typically0network0 forwarding0 runs0 in0kernel0space
Docker Networking Options
• Bridge – docker0• Open vSwitch• Port Mapping• Unix sockets / pipe
Many ways
Host
ContainerC
Container D Container E Container FContainer A Container B
DirectHostnetwork
Unix-domainsockets andother IPC
Docker0Linux bridge
Docker proxy (using iptables)
Open vSwitch
Port mapping
Option: Docker0 bridge
• Default network automatically created when no additional options “-net” or “-P” are specified• Each container is addressed by a static IP
address assigned by Docker• Similar to what we have as default with KVM or
VirtualBox• Host can reach container with IP on the bridge• But, outside traffic cannot reach the container
Nginx1172.17.0.18
C1172.17.0.19
C2172.17.0.20
172.17.42.1
Docker0 bridge
eth0 eth0 eth0veth002aa7a veth6df8377 veth7b0e4c6
eth0192.168.50.16
Docker0 bridge
Check Linux bridge ports and NAT rules under the hood# iptables –L+–t+nat -n
...
Chain'POSTROUTING'(policy'ACCEPT)
target'''''prot opt'source'''''''''''''''destination
MASQUERADE''all''DD 172.17.0.0/16''''''''anywhere
# sudo brctl show
bridge'name'''''bridge'id'''''''''''STP'enabled'''''interfaces
docker0'''''''''8000.56847afe9799'''no''''''''''''''veth002aa7a
veth6df8377
veth7b0e4c6
# docker inspect+--format='{{.NetworkSettings}}'+nginx1+
(See'for'yourself)
Port mapping• Provide access to the container from outside by
allocating a DNAT port in the range 49153-65535• Still uses Linux bridge docker0, but
adds iptables rules for the DNAT• In our example, nginx2 container is reachable by
accessing 192.168.50.16:49155
# docker+run+-P+-d+-i+--name=nginx2+-t+nginx
# iptables+–L+–t+nat+-n
...
Chain'DOCKER'(2'references)
target'prot'opt'source'''''destination
DNAT'''tcp''DD anywhere'''anywhere'''tcp'dpt:49155'to:172.17.0.19:80
DNAT'''tcp''DD anywhere'''anywhere'''tcp'dpt:49156'to:172.17.0.19:443
...
Host
nginx2 c1
172.17.42.1
Docker00bridge
eth0 eth0veth79ed06d veth6df8377
eth0192.168.50.16
Port mapping
For0the0new0nginx20container,0we0show0network0settings0below# docker inspect+nginx2
"NetworkSettings":'{"Bridge":'"docker0","Gateway":'"172.17.42.1","IPAddress":'"172.17.0.19","IPPrefixLen":'16,"MacAddress":'"02:42:ac:11:00:0f","PortMapping":'null,"Ports":'{
"443/tcp":'[{
"HostIp":'"0.0.0.0","HostPort":'"49157"
}],"80/tcp":'[
{"HostIp":'"0.0.0.0","HostPort":'"49158"
}]}}
Advanced:• It is possible to restrict the port
mapping to listen on specific host IP address and/or a specific host port number
• Use -p option as follows#+docker run+\–p+host_IP:host_port:container_port\–d+–i –t+nginx
HostGive full access of the host network to the container using --net=host option# docker run --net=host
--name=c3 -i –d –t base /bin/bash
Check network within container using ifconfig command through exec# docker exec c3 ifconfig eth0
eth0 Link encap:EthernetHWaddr 52:54:00:0d:3c:9finet addr:192.168.50.16Bcast:192.168.50.255
Host can talk to container using lo (localhost) interfaceContainers can listen on privileged ports (i.e., port numbers < 1024) of host
Give full access to network of a container XX to the new container YY using --net=container:XX option# docker run --net=container:nginx1
--name=c4 -i –d –t base /bin/bash
Check network within container using ifconfig command through exec# docker exec c4 ifconfig eth0
eth0 Link encap:EthernetHWaddr 02:42:ac:11:00:12inet addr:172.17.0.18Bcast:0.0.0.0
Container XX can talk to container YY using lo (localhost) interface
Container
Option: Open vSwitch
• Similar to Linux bridge, but different technology• Today, this is not the default with Docker• Allows programming with OVSDB and OpenFlow protocols
• Why? OpenvSwitch has many useful features!• VxLAN, GRE, VLAN based encapsulation and L2 forwarding• Encapsulation allows containers to pick any MAC/IP they want• Also possible to do L3 routing, ARP proxy etc, load-balancing• Access control, traffic rate limiting and prioritization• 10G/s or more packet processing throughput possible• 1) kernel, or 2) userspace, with optionally DPDK acceleration
Open vSwitch
A software virtual switch • Overlay Network• VxLan, GRE• Openflow
Dockerfile
What is dockerfile• A definition of image• FROM – base image from• RUN – initialize something• CMD – run after container created• EXPOSE – port • MAINTAINER – notification email send to• ADD – pass start.sh into container• VOLUME – path mapping• WORKDIR
#example'dockerfile D redisFROM ubuntuMAINTAINER ruifengyun "[email protected]"ADD ./start.sh /root/start.shRUN aptDget updateRUN aptDget install Dy redisDserverRUN aptDget install Dy opensshDserver#CMD redisDserver /etc/redis/redis.conf && /usr/sbin/sshdDDCMD ["redisDserver","/etc/redis/redis.conf"]EXPOSE 6379EXPOSE 22
# docker build Dt'trystack/redis
Demo page1FROM ubuntu:14.04MAINTAINER [email protected]
# Keep upstart from complainingRUN dpkgDdivert DDlocal DDrename DDadd /sbin/initctlRUN ln Dsf /bin/true /sbin/initctl
# Let the conatiner know that there is no ttyENV DEBIAN_FRONTEND noninteractive
RUN aptDget updateRUN aptDget Dy upgrade
# Basic RequirementsRUN aptDget Dy install mysqlDserver mysqlDclient nginx php5Dfpm php5Dmysql phpDapc pwgen pythonDsetuptools curl git unzip
# Wordpress RequirementsRUN aptDget Dy install php5Dcurl php5Dgd php5Dintl phpDpear php5Dimagick php5Dimap php5Dmcrypt php5Dmemcache php5Dming php5Dps php5Dpspell php5Drecode php5Dsqlite php5Dtidy php5Dxmlrpc php5Dxsl
# mysql configRUN sed Di De"s/^bindDaddress\s*=\s*127.0.0.1/bindDaddress = 0.0.0.0/" /etc/mysql/my.cnf
Demo page2# nginx configRUN sed Di De"s/keepalive_timeout\s*65/keepalive_timeout 2/" /etc/nginx/nginx.confRUN sed Di De"s/keepalive_timeout 2/keepalive_timeout 2;\n\tclient_max_body_size 100m/" /etc/nginx/nginx.confRUN echo "daemon off;" >> /etc/nginx/nginx.conf
# phpDfpm configRUN sed Di De "s/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/g" /etc/php5/fpm/php.iniRUN sed Di De "s/upload_max_filesize\s*=\s*2M/upload_max_filesize = 100M/g" /etc/php5/fpm/php.iniRUN sed Di De "s/post_max_size\s*=\s*8M/post_max_size = 100M/g" /etc/php5/fpm/php.iniRUN sed Di De "s/;daemonize\s*=\s*yes/daemonize = no/g" /etc/php5/fpm/phpDfpm.confRUN sed Di De "s/;catch_workers_output\s*=\s*yes/catch_workers_output = yes/g" /etc/php5/fpm/pool.d/www.confRUN find /etc/php5/cli/conf.d/ Dname "*.ini" Dexec sed Di Dre 's/^(\s*)#(.*)/\1;\2/g' {} \;
# nginx site confADD ./nginxDsite.conf /etc/nginx/sitesDavailable/default
# Supervisor ConfigRUN /usr/bin/easy_install supervisorRUN /usr/bin/easy_install supervisorDstdoutADD ./supervisord.conf /etc/supervisord.conf
Demo page3# Install WordpressADD http://wordpress.org/latest.tar.gz /usr/share/nginx/latest.tar.gzRUN cd /usr/share/nginx/ && tar xvf latest.tar.gz && rm latest.tar.gzRUN mv /usr/share/nginx/html/5* /usr/share/nginx/wordpressRUN rm Drf /usr/share/nginx/wwwRUN mv /usr/share/nginx/wordpress /usr/share/nginx/wwwRUN chown DR wwwDdata:wwwDdata /usr/share/nginx/www
# Wordpress Initialization and Startup ScriptADD ./start.sh /start.shRUN chmod 755 /start.sh
# private exposeEXPOSE 3306EXPOSE 80
CMD ["/bin/bash", "/start.sh"]
Docker Registry
Step 1 – Install Prerequisites
• Docker registry is a python application• Need some python libs
$ sudo apt-get update $ sudo apt-get -y install build-essential python-dev libevent-dev python-pip liblzma-dev
Step 2 – Install and Configure Docker Registry
sudo pip install docker-registry gunicorn --access-logfile - --debug -k gevent -b 0.0.0.0:5000 -w 1 docker_registry.wsgi:applicationcd /usr/local/lib/python2.7/dist-packages/docker_registry/lib/../../config/ sudo cp config_sample.yml config.ymlsudo mkdir /var/docker-registry
Questions?
Useful Links:http://blog.sequenceiq.com/blog/2014/06/17/ambari-cluster-on-docker/http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/