continuous deployment with kubernetes, docker and gitlab ci
TRANSCRIPT
![Page 1: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/1.jpg)
Continuous Deployment with Kubernetes, Docker
and GitLab CI
@alexander_kielClojure Berlin 2016
![Page 2: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/2.jpg)
![Page 3: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/3.jpg)
Outline• Continuous Deployment why?
• Docker
• Kubernetes
• Sample Clojure Service
• Deploy with GitLabCI
![Page 4: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/4.jpg)
Continuous Deployment• What do we want?
• Increase responsiveness
• Decrease time to market
• Gain confidence by deploying often in small amounts
• How to achieve that?
• Automate everything
• Always deploy the master into production
• Use feature toggles when needed
![Page 5: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/5.jpg)
Simple Git Workflow
• Works for in-house apps
• not for libs or shipping apps
• No versions, no tags, just SHA’s
• Latest commit on master is always deployed to production
• Feature/fix branches are merged when ready
masterfeature/fixbranches
1ebb95d
be61dda
6e4010d
![Page 6: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/6.jpg)
Docker• Like VM’s but much more light-weight and shippable
• Runs on Linux, executes processes in an isolated environment (resource limitation, filesystem, network)
• Container principle: Can contain everything, but looks the same from the outside
• A container platform can run every container
• Developers have max. freedom what to do
• In contrast: PaaS like Heroku - has to support the language
![Page 7: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/7.jpg)
Kubernetes• Container runtime platform
• Originally designed by Google - now Open Source
• One of the most active projects on GitHub - 20,000 stars, 40,000 commits, 15,000 issues, 200 releases
• Alternatives: Apache Mesos, Docker Swarm (lacks features)
![Page 8: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/8.jpg)
Kubernetes Architecturek8s-master-1
k8s-master-2
k8s-master-3
load-balancer-1
load-balancer-2
DNS RR
k8s-worker-1
proxy
app-1
k8s-worker-2
proxy
app-2
k8s-worker-n
proxy
app-k
etcd clusterquorum
HAProxy
• Runs on VMware ESX
• CoreOS Linux
• Single YAML file as configuration
• Everything in containers
![Page 9: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/9.jpg)
Kubernetes - Pods• A Pod is a deployable unit in
Kubernetes
• Pods can contain multiple containers
• Containers inside a Pod share on port space, can use localhost and can communicate via IPC and shared memory
• Idea: one process per container - many cooperating processes in one Pod
apiVersion: v1 kind: Pod metadata: name: <pod-name> labels: <key>: <value> spec: containers: - name: <container-name> image: <container-image> ports: - containerPort: 80 env: - name: <key> value: <value>
![Page 10: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/10.jpg)
Kubernetes - Deployments
• A Deployment ensures that certain number of Pods are always running
• It consists of a Pod template and the number of replicas
• It supports hot-redeployments by changing parts of the Pod template
• Horizontal scaling is possible
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: <deployment-name> spec: replicas: 2 template: metadata: labels: <key>: <value> spec: containers: - name: <container-name> image: <container-image> ports: - containerPort: 80 env: - name: <key> value: <value>
![Page 11: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/11.jpg)
Kubernetes - Services• Kubernetes uses an overlay
network to provide different address spaces (we use flannel)
• Every Pod has an IP address - but it changes every time one is created
• Services provide a stable IP address for groups of Pods
• Service names are resolvable by an internal DNS
• Service selectors are used to match Pods according to there labels
apiVersion: v1 kind: Service metadata: name: clojure-berlin-2016 labels: app: lens spec: type: NodePort ports: - port: 80 targetPort: 80 protocol: TCP selector: service: clojure-berlin-2016
![Page 12: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/12.jpg)
Kubernetes - External Access• Kubernetes networks are internal
only
• External access through load balancers necessary
• Certain Platforms like Google Compute Engine provide load balancer integration with Kubernetes
• We have our own solution as a combination of HAProxy and Kubernetes NodePort
• Kubernetes Services with type NodePort are exposed on every worker under a certain port
frontend http bind 0.0.0.0:80 mode http option httplog
acl host_clj hdr(host) clj.<domain>
use_backend clj if host_clj
backend clj mode http balance roundrobin option httplog server worker-1 <ip>:32599 check server worker-2 <ip>:32599 check
![Page 13: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/13.jpg)
Deployment Lifecycle
GitLab CI
Source Code
build
test
Kubernetes Test
Cluster
Kubernetes Prod
Cluster
automatic deployment
manualdeployment
git push
![Page 14: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/14.jpg)
Sample Clojure Service• .gitlab-ci.yml
• Like .travis.yml contains instructions for GitLabCI how to test, build and deploy
• Dockerfile
• Instructions for Docker how to build the image of the app
• Artifact of the build is a docker image - not uberjar
• kube-deployment.yml
• Kubernetes deployment instructions
• kube-svc.yml
• Kubernetes service description
https://github.com/alexanderkiel/clojure-berlin-2016
![Page 15: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/15.jpg)
The Core Namespace
(ns clojure-berlin-2016.core (:require [aleph.http :as http] [clojure.core.async :refer [<!! chan]]))
(defn -main [& args] (-> (fn [_] {:status 200 :body "Clojure Berlin 2016"}) (http/start-server {:port 8080})) (<!! (chan)))
• A simple web server returning "Clojure Berlin 2016"
![Page 16: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/16.jpg)
The Leiningen Project File
(defproject clojure-berlin-2016 "<VERSION>" :dependencies [[aleph "0.4.1"] [org.clojure/clojure "1.8.0"] [org.clojure/core.async "0.2.395"]]
:main clojure-berlin-2016.core)
• <VERSION> is replaced at build time by the Git SHA
• :main is for lein run to work
![Page 17: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/17.jpg)
.gitlab-ci.yml - test/buildimage: clojure:lein-2.7.1
stages: - test - build - deploy
test: stage: test tags: - docker script: - lein test
build: stage: build tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" project.clj - docker build -t clojure-berlin-2016:$CI_BUILD_REF . - docker push clojure-berlin-2016:$CI_BUILD_REF
![Page 18: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/18.jpg)
.gitlab-ci.yml - deploy branchdeploy-branch: stage: deploy environment: test image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-test - kubectl apply -f kube-deployment.yml except: - master when: manual
• Used to test a feature/fix branch in a full environment
![Page 19: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/19.jpg)
.gitlab-ci.yml - deploy test
deploy-master: stage: deploy environment: test image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-test - kubectl apply -f kube-deployment.yml only: - master
![Page 20: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/20.jpg)
.gitlab-ci.yml - deploy prod
deploy-prod: stage: deploy environment: prod image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-prod-a - kubectl apply -f kube-deployment.yml only: - master when: manual
![Page 21: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/21.jpg)
Docker fileFROM clojure:lein-2.7.1
COPY src /app/src COPY project.clj /app/
WORKDIR /app
RUN lein with-profile production deps
EXPOSE 80
CMD ["lein", "with-profile", "production", "run"]
• Just copy the sources into the container
• Use Leiningen itself to run in production
![Page 22: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/22.jpg)
kube-deployment.ymlapiVersion: extensions/v1beta1 kind: Deployment metadata: name: clojure-berlin-2016 spec: replicas: 2 template: metadata: labels: app: lens service: clojure-berlin-2016 spec: containers: - name: clojure-berlin-2016 image: dreg.life.uni-leipzig.local/clojure-berlin-2016:<VERSION> ports: - containerPort: 8080 resources: requests: cpu: "125m" memory: "1Gi" limits: cpu: 1 memory: "2Gi"
![Page 23: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/23.jpg)
kube-svc.yml
apiVersion: v1 kind: Service metadata: name: clojure-berlin-2016 labels: app: lens spec: type: NodePort ports: - port: 80 targetPort: 8080 protocol: TCP selector: service: clojure-berlin-2016
![Page 24: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/24.jpg)
Steps to Follow• Create the Kubernetes Service
• kubectl create -f kube-svc.yml
• Edit HAProxy Config
• add rules and backend for the service
• Push to GitLab
• git push
![Page 25: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/25.jpg)
Pipeline in GitLab CI
![Page 26: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/26.jpg)
Deployment in GitLabCI
![Page 27: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/27.jpg)
Environments in GitLabCI
• Very good visibility of wich commit is deployed in which environment right now
• Manual deployment to prod possible
![Page 28: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/28.jpg)
Environment History
• Easy to see when what commit was deployed
• Rollback possible
![Page 29: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/29.jpg)
Numbers• Our team has 4 developers
• We run 2 Kubernetes clusters (test and prod) with about 96 GB RAM and and 24 vCPU’s each
• We run about 60 pods in production
• We have other services like central log aggregation running using Fluentd and Elasticsearch/Kibana
![Page 30: Continuous Deployment with Kubernetes, Docker and GitLab CI](https://reader034.vdocuments.mx/reader034/viewer/2022042507/5871adc21a28abda6a8b5fbd/html5/thumbnails/30.jpg)
Thank You• Sample Project on Github
https://github.com/alexanderkiel/clojure-berlin-2016
• Twitter@alexander_kiel