from dev to devops · from dev to devops: an unexpected journey luis angel vicente sanchez...

From Dev to DevOps:

An Unexpected Journey

Luis Angel Vicente Sanchez

BigCentech Ltd

[email protected]

From Zero to Hero?

2

What is DevOps?

3

By Devops.png: Rajiv.Pantderivative work: Wylve - This file was derived from Devops.png:, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=20202905

What is DevOps?

4

Source

Automation

Configuration

Management Logging

Monitoring

Alerting

What is DevOps?

5

Source

Automation

Infrastructure

as Code Logging

Monitoring

Alerting

What is DevOps?

6

Who am I?

7

12+ Software Engineer

Big fan of functional programming

Lately a lot of Big DataLuis Angel Vicente Sanchez

[email protected]

http://bigcente.ch

What am I doing here?

8

What am I doing here?

9


10


11


12


13

Deploying to AWS

14

Deploying to AWS: Terraform

15

Deploying to AWS: Terraform modules

16

https://github.com/heartysoft/cassy-up/tree/master/terraform/dse-cassandra

https://github.com/heartysoft/cassy-up/tree/master/terraform/kafka

Deploying to AWS: EC2 instances

17

resource "aws_instance" "dse-cassandra" {

count = "${var.count}"

ami = "${var.ami_id}"

key_name = "datalake"

instance_type = "${var.instance_type}"

availability_zone = "${var.availability_zones[...]}"

subnet_id = "${var.subnet_ids[var.availability_zones[...]]}"

vpc_security_group_ids = [ "${var.sg_ids}" ]

tags = {

Name = "${format("%s-%d", var.tag_name, count.index + 1)}"

}

}

Deploying to AWS: EBS volumes

18

resource "aws_ebs_volume" "dse-cassandra-ebs-data" {


availability_zone = "${var.availability_zones[...]}"

size = "${var.data_ebs_size}"

type = "${var.data_ebs_type}"

tags {

Name = "${var.tag_name} - Data Vol"

}

}

Deploying to AWS: EBS attachments

19

resource "aws_volume_attachment" "dse-cassandra-ebs-data-attach" {


device_name = "/dev/xvdh"

volume_id = "${aws_ebs_volume.dse-cassandra-ebs-data.*.id[count.index]}"

instance_id = "${aws_instance.dse-cassandra.*.id[count.index]}"

force_detach = true

}

Deploying to AWS: Provisioning

20

resource "null_resource" "provision-dse-cassandra" {

...

connection {

user = "${var.ssh_user}"

host = "${element(aws_instance.dse-cassandra.*.private_ip, count.index)}"

agent = true

private_key = "${file("${var.ssh_identity_file}")}"

}

...

}


21


...

provisioner "file" {

source = "${var.provisioning_scripts}/dse-cassandra"

destination = "/tmp/cassy-up"

}

provisioner "file" {

source = "${var.provisioning_scripts}/environments/aws/dse_cassandra.sh"

destination = "/tmp/cassy-up/dse_cassandra.sh"

}

...

}


22


...

provisioner "remote-exec" {

inline = [

"echo 'export CASSANDRA_CLUSTER_NAME=${var.cluster_name}' ...",

...

"chmod -R a+x /tmp/cassy-up/*",

"sudo /tmp/cassy-up/dse_cassandra.sh"

]

}

}

Kubernetes

23

Kubernetes: Why?

24

Self-healing

Service

Discovery

Horizontal

Scaling

Rolling

upgrades

and

rollbacks

Kubernetes: Resize the cluster

25

KOPS


26

┌[luis@BigCentech-Precision5520]-(~)

└> kops create cluster --zones eu-west-1a,eu-west-1b,eu-west-1c $CLUSTERNAME


└> kops edit cluster $CLUSTERNAME


└> kops update cluster $CLUSTERNAME --yes


└> kops export kubecfg $CLUSTERNAME


27


└> kops edit ig nodes


└> export KOPS_FEATURE_FLAGS="+DrainAndValidateRollingUpdate"


└> kops rolling-update cluster --yes

Kubernetes: Deploying

28

kubectl

Pods Services

Kubernetes: What is a Pod?

29

kubectl create -f

mypod.yaml

apiVersion: v1

kind: Pod

metadata:

name: nginx

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx

ports:

- containerPort: 80

Kubernetes: What is a Service?

30

kubectl create -f

myservice.yaml

apiVersion: v1

kind: Service

metadata:

name: nginx-service

labels:

app: nginx

spec:

ports:

- port: 80

targetPort: 80

selector:

app: nginx

Kubernetes: What is a Deployment?

31

kubectl create -f

mydeployment.yaml

apiVersion: apps/v1beta1

kind: Deployment

metadata:

name: nginx-deployment

spec:

replicas: 3

template:

metadata:

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx

ports:

- containerPort: 80

Kubernetes: How to debug?

32

12 Factor App


33


34

Node 0

fluentd

Node 1

fluentd

Node 3

fluentd

ES

CLUSTERKIBANA

Kubernetes: Problems?

35


36

HEAPSTE

R


37

HEAPSTER INFLUXDB

Node 0

Pod 1

Pod 0

Node 1

Pod 3

Pod 2

Node 3

Pod 5

Pod 4

GRAFANA


38

TELEGRAF INFLUXDB

Node 0

Pod 1

Pod 0

Node 1

Pod 3

Pod 2

Node 3

Pod 5

Pod 4

GRAFANA

Deployment Pipeline

39

Deployment Pipeline: Gitlab CI

40

Deployment Pipeline: Stages and Jobs

41

Deployment Pipeline: Stages

42

Build

(on commit)

Test

(on commit)

Deploy

(on tag)Publish ContainerBuild Test Deploy


43

stages:

- build

- test

- publish-container

- deploy

Deployment Pipeline: Jobs

44

build:

stage: build

script:

- npm install

- npm run build

publish-container:

stage: publish-container

image: heartysoft/docker-builder-aws:0.15.0

script:

- chmod +x ./ci/*

- ./ci/publish-container.sh "app-ui" "$CI_COMMIT_TAG" "app/app-ui"

only:

- tags

Deployment Pipeline

45

Publish Container

(on tag)

Build

(on commit)

Test

(on commit)

Deploy

(on tag)

Publish Container

(on tag)

Build

(on commit)

Test

(on commit)

Deploy

(on tag)

Deployment Pipeline: How to deploy?

46

kubect

l

Deployment Pipeline: How to deploy?

47

kubect

l

Deployment Pipeline: Helm

48

Deployment Pipeline: Helm Chart

49

└── mychart

├── charts

├── Chart.yaml

├── templates

│ ├── deployment.yaml

│ ├── _helpers.tpl

│ ├── ingress.yaml

│ ├── NOTES.txt

│ └── service.yaml

└── values.yaml

Deployment Pipeline: Chart.yaml

50

apiVersion: v1

description: A Helm chart for Kubernetes

name: mychart

version: 0.1.0

Deployment Pipeline: values.yaml

51

replicaCount: 1

image:

repository: nginx

tag: stable

pullPolicy: IfNotPresent

service:

name: nginx

type: ClusterIP

externalPort: 80

internalPort: 80

resources: {}

Deployment Pipeline: templates/service.yaml

52

apiVersion: v1

kind: Service

metadata:

name: {{ template "fullname" . }}

labels:

app: {{ template "name" . }}

chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}

release: {{ .Release.Name }}

heritage: {{ .Release.Service }}

spec:

type: {{ .Values.service.type }}

ports:

- port: {{ .Values.service.externalPort }}

targetPort: {{ .Values.service.internalPort }}

protocol: TCP

name: {{ .Values.service.name }}

selector:

app: {{ template "name" . }}

release: {{ .Release.Name }}

Deployment Pipeline: Deploy from filesystem

53


└> helm upgrade --install -f dev-values.yaml myapp ./mychart

Deployment Pipeline: Helm Repo

54

└── repo

├── index.yaml

├── myapp-1.0.0.tgz





└── myapp-1.3.0.tgz

Deployment Pipeline: Helmet

55

https://hub.docker.com/r/daemonza/helmet

/

https://github.com/daemonza/helmet

Deployment Pipeline: Deploy from helm repo

56


└> helm upgrade --install --version 1.2.3 -f dev-values.yaml myapp helmet/mychart


57

Publish Container

(on tag)

Build

(on commit)

Test

(on commit)

Deploy

(on tag)

Publish Chart

(on tag)

Questions?

58

THANK YOU!

from dev to devops · from dev to devops: an unexpected journey luis angel vicente sanchez...

Documents