DevOps:

Cloud-inspired,

On-Premises

Patrick Gemme (pgemme@holycross.edu)

Rick Nickle (rnickle@holycross.edu)

Introductions

Patrick Gemme

● Network Engineer

● Windows Server

● Hyper-V

Rick Nickle

● Systems Integration

● Linux

● QA and Test

Plan of Presentation

● Cultural issues with migrating some applications from on-

premises to cloud suggested a DevOps approach.

● Automate our applications environment for remaining on-

premises applications, by leveraging our Cloud approach.

● Measure and monitoring our environment in order to

continuously improve our availability, performance and

capabilities.

● Share our experience.

Why Cloud-inspired?

● Cultural issues inspired our migration of apps to the Cloud

using DevOps principles.

● Automation of on-premises application deployment was built

on our success with the Cloud migration.

● Measurement is the key element which we hope to

continuously improve as we extend our embrace of DevOps.

● Sharing lessons learned, both positive and negative, with

our own team and our peers is part of the open culture we

are promoting.

Cloud

Cloud Applications Environment

● Previously hosted on-premises, moved to

Acquia and WPEngine cloud services.

● Both providers use Git to manage

content.

● Only one provided a full-featured

dashboard for managing releases.

Cloud Applications Strategy

Challenge Response Toolsets

Dev. and Comm. not familiar

with revision control and/or

Git.

Find usability tools and

develop training for release

procedures.

Git-bash, GitLab, GitKraken,

slides and Panopto videos.

Dev. and Comm. step on each

others releases.

Require change control

approval before release.

SA Manage ticketing using

REST API, integrated to

GitLab CI/CD.

Dev. and Comm. not testing or

not testing on environment

equivalent to production.

Require release to staging

environment before release to

production.

Use Merge Request to require

release to test before

production can release.

Definition of CI/CD

Continuous Integration

● Any authorized user

can checkout,

modify, checkin, and

tag code at any time.

Continuous Deployment

● Any authorized user

may use CI to trigger

automation, including

release to test or

production.

Authorization for CI/CD

● Each project has an authorization

dashboard.

● Project owner can control level of

authorization.

● Test branches are not protected.

● Production branches are protected○ Use Merge Request of tested branch to update.

Interface for CI/CD

● Accessed through git transactions, the

dashboard, or the REST API.

● All transactions are browsable.

● Webhooks are included for social media and

other apps. (e.g., Slack)

Enable an Application for Cloud

Two requirements:

● App Code

● A CI/CD file○ A simple YAML

file that defines

how to release.

code

CI/CD

.gitlab-ci.yml

An overview of Cloud CI/CD file

production:

script:

- /ci-tools/bin/check_samanage --tag $CI_BUILD_TAG

- /ci-tools/bin/ci_target --verbose

only:

- /^PROD-SA.*$/

except:

- branches

Cloud Release Flow

● CI/CD file defines release

flow.

● Pre-built Container has code

and credentials for release to

cloud.

● User checks in code and tags

code.

● CI/CD invokes Docker to

checkout code and push it to

Cloud repo.

code

Checkin

& Tag

Docker Container

Checks Code Out

Docker Container

Pushes Code to

Cloud

Cloud and CI/CD

prod

testPropose

Change Control

Ticket?

CI/CD

CI/CDTag

“TEST”

Tag

“PROD”

Create

Branch

yes

no

Branch

“A”

Branch

“master”

...coding..

.

branching tagging releasing runningthinking

merging

Applications

Legacy Applications Environment

● Virtualization environment must be shut down when

host node is patched.

● Release by many methods: sftp, Capistrano, rsync

● Release requires privileged access

● Test and Development not equivalent to production

● Single app environments for each group of application:

rails3, rails4, php5, java, tomcat.

Consolidation, Release Management

● Asked to consolidate applications and

restrict deployment ability, preferably

behind automation controlled via CI/CD.

● Used a simple rsync based prototype to

release code fragments and “gadgets.”

Refactor for Microservices and CI/CD

● Experiments with consolidating small

applications led to trying to containerize

medium sized LAMP applications.

● LAMP applications performed very well

and were relatively easy to port, quickly

put several into DevOps production.

Microservices, CI/CD Strategy

Challenge Response Toolsets

Application tied to server

configuration; Disparity

between Test and Prod

Containerize Applications Docker Swarm

Release requires privilege Authorization defined in

application repository.

GitLab CI/CD

Consolidate similar services Standardized configurations,

containers

DSC, Docker Swarm

Legacy VM Environment

● Legacy application environment, older VM technology.

● Variations in server configurations over time.

● Security standards not defined for original deployment.

● Consolidate similar services.

● Automate deployment of applications (non-privileged)

● Virtualization host not clustered.

Infrastructure as Code/Data StrategyChallenge Response Toolsets

Virtual servers unavailable

when host patched or

rebooted.

Hyperconverged Clusters Hyper-V, CSV/Storage Spaces

Variations in VM configs. VM templates, managed

configurations.

SaltStack, DSC

Security and best practices Published standards Group Policies, DSC

Infrastructure as Code/Data

● DSC not only encodes the instructions for recreating,

replicating and scaling the environment, it is also a

design document that can be referred to for

understanding, auditing and measuring the

environment.

● Virtualization provides an API for manipulating the

running environment.

Containerize an Application with DevOps

You need four things:

● Application code

● Dockerfile

● CI/CD file

● Compose file

codeDockerfile

Compose.ymlCI/CD

.gitlab-ci.yml

Container Release Flow

● CI/CD file defines Dockerfile and

Compose file to use for release.

● Dockerfile defines how to build

container from code.

● Compose file describes how to

launch container.

● User checks in and tags code.

● CI/CD invokes Docker to

checkout code, build container

with Dockerfile and launch with

Compose file.

code

Checkin

& Tag

Docker Container

Checks Code Out

Docker Container

builds and launches

new container.

Overview of Container CI/CD File

HCREG=docker-reg.holycross.edu

PROJECT=$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME

COMPOSE=docker-compose-$CI_JOB_NAME.yml

DEPLOY_OPTIONS=”--resolve-image always --with-registry-auth”

prod:

script:

- CONTAINER=$HCREG/$PROJECT:$CI_JOB_NAME

- /ci-tools/bin/check_samanage --tag $CI_BUILD_TAG

- docker build --pull --build-arg APPDEPLOY=$CI_JOB_NAME -t $CONTAINER .

- docker push $CONTAINER

- docker stack deploy -c $COMPOSE $CI_PROJECT_NAME $DEPLOY_OPTIONS

Challenges and OpportunitiesChallenge/Opportunity Response Toolsets

Some applications have very

long container build times,

slowing development cycle.

Consider alternative

development pipelines

Hybrid scripts? Jenkins?

Capistrano?

Collaboration across

organizational lines.

Encourage more of this

(DataOps)

Social media, word of mouth,

training.

Operational challenges slow

adoption: legacy, priorities,

failures.

First things first. Patience, reflection, and

lessons learned.

Monitoring

Scaling, Performance, AvailabilityChallenge Response Toolsets

Apache MPM overwhelmed by

peak loads

Change from ‘prefork’ to

‘event’, multiple servers,

ingress buffering and caching,

functional monitoring.

Apache 2, Cloudflare, Docker

Stack, SCOM, Pingdom

Current monitoring solution

unable to provide useful deep

website monitoring

Actually got help from our

security team; setup web-

based test logins

Pingdom

Monitoring and Culture

● Making decisions based

on measurements.

● If you monitor something and alert on it

but don’t have a defined response, don’t

monitor it.

Monitoring & Automation

• Common problems with common fixes

should be automated.

• Most troubleshooting starts with basic tasks

which can be auto-launched.

• Ticketing system flow and auto ticket

creation (API’s, scheduled, form filled).

Monitoring & Automation (cont.)

Common Issue

Microsoft SCOM

Automated Fix

Sharing

Our DevOps OutlookStrength Weakness Threat Opportunity

CI/CD encourages frequent

releases of enhancements,

“blameless” culture.

Basic monitoring is effective,

difficulty measuring non-

functional requirements.

Need to maintain containers

for performance and

security: DevSecOps.

Containers have built-in

monitoring hooks for

applications: stateful apps.

DSC, templates and

containers enhance

availability, DR and

scalability: infrastructure

as code/data.

Sharing of methods, tools

and results are uneven.

Build new apps with

DevOps in mind.

Docker access by

developers is privileged

access, restrict Docker

access to test.

DevSecOps.

New model for deployment

requires refactoring of

application: 12-factor app

model.

Adoption of DevOps for

some legacy tools impeded

by vendor defined

environments. LegacyOps.

Success Stories I

● When marketing team, who had

previously complained about complexity

of process, commented that they found it

empowering after getting used to it.

Mixed Results I

● When customer in another department

who previously had delegated all changes

to operations, was trained and given

access to release configuration changes

themselves, was suddenly eager to move

from on-premises to cloud.

Success Stories II

● When development team member

commented that they would like to clone an

existing application as a baseline for a new

application.

● When development team member built and

tested an application themselves by creating

a new project from scratch.

Cautionary Tale II

● When developer points out that his Ruby

release mechanism was faster albeit with

more maintenance overhead, while

acknowledging that the Docker based

mechanism has superior performance

and scaling features.

Success Stories III

● When marketing team member who

previously had thought version control

and change control was a waste of time,

commented that the development team

wasn’t properly checking in, tagging and

testing their code before release.

DevOps Toolsets To Explore● GitLab Features

○ Build Artifacts (deliverables consumed by applications)

○ Static testing (code)

○ Environments (serverless, Kubernetes)

● QA○ Locust (load testing)

○ Katalon (app testing)

○ Robot Framework (acceptance testing)

● Monitoring and Measurement○ ELK (syslog processing)

○ Grafana/Prometheus (metrics and visualization)

● Load Balancing○ Traefik (container based application load balancer)

● Cloud-like user portal (IaaS, SaaS, PaaS)○ Azure Stack, Nutanix Calm, Heroku

Q & A

Panel at 3pm

Notes

Reference

Strategic Overview for PresentationInfrastructure/Operations

● Monitoring (SCOM)

● Management (Salt)

● Group Policy (AD)

● Containers (Docker)

● Load Balancers (Apache)

● Database Servers

(MySQL/Oracle)

Applications/Development

● Continuous Integration

(GitLab)

● Continuous Deployment

(GitLab)

● Source Management (GitLab)

● QA/Test

● Twelve Factor

DevOps Integration

Token Funny Quote