ùcu Ç|& ëÊ· n o=ÔûbüÜxyÎ ÈÄi¸ g1 k · 2017-11-06 · 1.8 infrastructure as code...

DevOps Fundamentals

Web Age Solutions Inc.USA: 1-877-517-6540Canada: 1-866-206-4644Web: http://www.webagesolutions.com

The following terms are trademarks of other companies:

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

IBM, WebSphere, DB2 and Tivoli are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

Other company, product, and service names may be trademarks or service marks of others.

For customizations of this book or other sales inquiries, please contact us at:

USA: 1-877-517-6540, email: [email protected]: 1-866-206-4644 toll free, email: [email protected]

Copyright © 2015 Web Age Solutions Inc.

This publication is protected by the copyright laws of Canada, United States and any other country where this book is sold. Unauthorized use of this material, including but not limited to, reproduction of the whole or part of the content, re-sale or transmission through fax, photocopy or e-mail is prohibited. To obtain authorization for any such activities, please write to:

Web Age Solutions Inc.439 University AveSuite 820TorontoOntario, M5G 1Y8

Table of ContentsChapter 1 - DevOps Introduction.........................................................................................5

1.1 Dev and Ops Views...................................................................................................51.2 Leading By Example ................................................................................................51.3 What is DevOps?.......................................................................................................51.4 More DevOps Definitions.........................................................................................61.5 DevOps and Software Delivery Life Cycle..............................................................61.6 Main DevOps' Objectives.........................................................................................61.7 The Term "DevOps" is Evolving!.............................................................................71.8 Infrastructure as Code...............................................................................................71.9 Agile IT in the Cloud ..............................................................................................81.10 DevOps on the Cloud..............................................................................................81.11 Prerequisites for DevOps Success ..........................................................................91.12 Alignment with the Business Needs.......................................................................91.13 Collaborative Development .................................................................................101.14 Continuous Testing and Integration......................................................................101.15 Continuous Release and Deployment ..................................................................101.16 Continuous Application Monitoring.....................................................................111.17 Summary...............................................................................................................11

Chapter 2 - Standing Up DevOps .....................................................................................132.1 Standing Up DevOps...............................................................................................132.2 Things to Look For and Avoid................................................................................132.3 IT Assets Ownership...............................................................................................142.4 Viewing Applications As Products, not Projects....................................................142.5 DevOps in the Enterprise........................................................................................152.6 IT Governance ........................................................................................................152.7 Governance and Risk Mitigation............................................................................162.8 DevOps Adoption Steps..........................................................................................162.9 DevOps Adoption Steps..........................................................................................172.10 DevOps Adoption Steps........................................................................................172.11 DevOps Adoption Steps........................................................................................172.12 Select DevOps Techniques and Practices.............................................................182.13 Select DevOps Techniques and Practices.............................................................182.14 Select DevOps Techniques and Practices.............................................................192.15 Select DevOps Techniques and Practices.............................................................192.16 Select DevOps Techniques and Practices.............................................................202.17 Service Quality Metrics.........................................................................................212.18 Summary...............................................................................................................22

Chapter 3 - DevOps Tools.................................................................................................233.1 The Choice of Cloud Platform ...............................................................................233.2 IaaS for DevOps......................................................................................................233.3 PaaS for DevOps.....................................................................................................243.4 Containerization Tools............................................................................................243.5 System Configuration Automation and Management.............................................253.6 System Configuration Automation and Management.............................................25

3.7 Continuous Integration (CI) Systems......................................................................263.8 Build and Dependency Management Systems........................................................273.9 Build and Dependency Management Systems........................................................273.10 Select DevOps Tools.............................................................................................283.11 Collaborative Lifecycle Management Solutions from IBM..................................283.12 The Collaborative Lifecycle Management Diagram.............................................293.13 The IBM Collaborative Lifecycle Management Platform ..................................293.14 Rational Team Concert (RTC)..............................................................................293.15 Rational Quality Manager (RQM)........................................................................303.16 Rational DOORS Next Generation (DNG)...........................................................313.17 Summary...............................................................................................................31

Chapter 4 - Containerization Systems Overview...............................................................334.1 Virtualization...........................................................................................................334.2 Hypervisors.............................................................................................................334.3 Hypervisor Types....................................................................................................344.4 Type 1 hypervisors..................................................................................................344.5 Type 2 hypervisors .................................................................................................344.6 Type 1 vs Type 2 Processing..................................................................................354.7 Paravirtualization....................................................................................................364.8 Virtualization Qualities (1/2)..................................................................................364.9 Virtualization Qualities (2/2)..................................................................................364.10 Disadvantages of Virtualization............................................................................374.11 Containerization....................................................................................................374.12 Virtualization vs Containerization........................................................................384.13 Where to Use Virtualization and Containerization...............................................384.14 Popular Containerization Systems .......................................................................384.15 What are Linux Containers...................................................................................394.16 Docker ..................................................................................................................394.17 OpenVZ.................................................................................................................404.18 Solaris Zones (Containers)....................................................................................404.19 Summary...............................................................................................................40

Chapter 1 - DevOps Introduction

ObjectivesKey objectives of this chapter

DevOps introduction

Business value of DevOps

Standing up DevOps capability

1.1 Dev and Ops Views

The Dev View The Ops View • We have aggressive deadlines --

Business is all over us• Ops are much too sluggish supporting

us (provisioning integration environment, etc.)

• They lost the application zip file we emailed them yesterday night -- it was eventually found in their "junk mail" folder

• Overall, we don't have trust and confidence in Operations (Ops) -- they are more like Oops, then Ops

• Dev is all over us• The application set-up guide sent by Dev

was not complete -- they missed some critical steps, which resulted in our wasted time

• With so many new applications being released in the environment, we can no longer guarantee uninterrupted services

• Overall, we don't trust Dev

1.2 Leading By Example ...

1.3 What is DevOps?

DevOps is short for Development and Operations

It is an approach to delivering software solutions in a continuous manner


based on lean (minimizing waste of resources, reducing number of defects, etc.) and agile practices

DevOps help manage complexities of Enterprise applications by creating a collaborative environment with participants coming not only from Development and Operations, but also from Business, QA, and other stakeholder groups

◊ In other words, DevOps is not only about Development and Operations!

The DevOps practice has been popularized by organizations adopting the Cloud-as-a-Service computing model

1.4 More DevOps Definitions

You can view DevOps as

◊ a culture, or

◊ a cross-team software delivery discipline (paradigm)

that tries to reconcile competing perspectives (e.g. those of Dev vs Ops) and promote collaboration by stepping over the silos of isolated, group-centric interests

1.5 DevOps and Software Delivery Life Cycle

To efficiently increase the velocity of application delivery, DevOps activities span the whole software delivery life cycle (not only its deployment!):

◊ Development

◊ Testing

◊ Deployment

◊ Operation

1.6 Main DevOps' Objectives

Continuous software delivery planning and control

6


Software delivery processes optimization

Software delivery process consistency and predictability

Minimization of the number of software defects and unnecessary re-work

Software delivery cycle time reduction

Notes:

Planning is viewed by some developers as an unnecessary overhead hampering their "real work"; those developers rely on tribal knowledge of some opaque team processes they establish themselves. While it may work in a short-term perspective, the lack of transparency and overall planning across various stakeholder groups would normally result in problems with project delivery sustainability at faster rates.

1.7 The Term "DevOps" is Evolving!

Originally, DevOps was used to refer to the practice adopted by Operations to borrow some of the tools and processes used in software development

◊ For example:

Admin scripts were placed in a version control system

Now DevOps is used in the context of the shared responsibility between Development and Operations for delivering high quality software products

◊ For that, where appropriate, the reporting lines are merged and simplified

◊ Development goals (introduce code, configuration, etc. changes to the system) are aligned with those of Operations (maintain the target system stability)

DevOps nowadays is becoming more embracing being not only about Tools, but also about People and Processes

1.8 Infrastructure as Code

"Infrastructure as Code" is a practice of provisioning infrastructure by executing system management and configuration scripts

7


Under DevOps, Dev is granted system administration privileges to run the infrastructure set-up scripts to automatically provision the necessary development and testing environments

◊ Provisioning of other environments (staging, production) may still be the exclusive prerogative of personnel in the DevOps' Operations role

Infrastructure as Code is effectively supported by such tools as Chef, Puppet, and IBM UrbanCode Deploy

1.9 Agile IT in the Cloud

Cloud facilitates agile IT practices that allow businesses to quickly respond to market forces

IT responds to rapidly changing business requirements by creating a cloud-based execution environment that allows effective and optimized reuse of existing services through

◊ re-configuration,

◊ re-composition, and

◊ introducing new services that also lend themselves to composition and reuse

1.10 DevOps on the Cloud

Some of the more important activities performed by Operations are related to provisioning computing resources

DevOps agility can been dramatically enhanced with adopting Cloud Computing

◊ Developers can easily self-provision the needed resources (virtual servers, extra disk storage, back-end systems, etc.)

◊ Many cloud platforms allow for application code snapshotting which can be used for environment replication / cloning (Dev → QA → UAT → Prod)

This feature also facilitates defect reproduction

8


◊ In the cloud, it is much easier to quickly set up and tear down "Production-like" systems at a minimal cost

1.11 Prerequisites for DevOps Success

A good relationship between Dev and Ops is necessary but not sufficient for the overall success of DevOps operations

DevOps success depends on the proper execution of all the technical aspects of the Software Development Life Cycle (SDLC) phases and steps established in the organization

◊ Sometimes, the existing SDLC documentation needs to be adjusted to make it aligned with the agile practices used by DevOps

The following elements and capabilities need to be put in place for DevOps to meet its objectives:

◊ Alignment with the business needs

◊ Collaborative development

◊ Continuous testing and integration

◊ Continuous release and deployment

◊ Continuous application monitoring

1.12 Alignment with the Business Needs

In essences, DevOps is a business-driven software solutions delivery process

The DevOps practice is instrumental in reliably materializing a business idea in a software product, ultimately delivering value to customers

DevOps processes improve product time-to-market metric (faster time to value) and enable organizations to react to new market demands more quickly

With DevOps, customer feedback on product is captured and quickly incorporated in the next iteration of the product delivered in a continues manner

9


◊ The ultimate result of a fast accommodation of the customer feedback is an enhanced customer experience, customer loyalty, and larger market share

1.13 Collaborative Development

High quality software development is predicated on the collaborating development practices, including:

◊ Development teams work is done in accordance with established code standards, styles, and living centralized developer documentation (wiki pages, etc.)

Development teams may be geographically dispersed or formed at the last moment, making the above practice indispensable

◊ The stewards of the target system's architecture and design are cooperating with development to quickly accommodate any design flaws / gaps identified during development

1.14 Continuous Testing and Integration

Development of discrete software components must go hand-in-hand with the development and application of appropriate unit tests as prescribed by the Test-Driven Development (TDD) process

◊ TDD is an agile software development practice

Integration of software components must start as early as possible (even though the work on components may not yet been fully completed) and conducted frequently (sometimes, several times a day)

◊ This process is known as the Continuous Integration (CI) agile practice which helps with catching integration problems early

1.15 Continuous Release and Deployment

Deployment has always been one of the primary activities of Operations

◊ With the advent of the Cloud Computing, Development can take over some workload of application deployment and perform self-provisioning

10


of cloud computing resources they require

To support reliable continuous releases, the deployment process must be automated; any failed deployments must be rolled back in its entirety in an atomic operation without affecting applications currently running in production

Deployment parameters, such as average deployment time, size of the deployment bundle, etc., must be recorded and kept for reference

1.16 Continuous Application Monitoring

Application run-time behavior monitoring should begin in production-like environments where the application would have setup, configuration, and other parameters close to those used in production

Things to look for:

◊ Run-time application behavior (CPU, RAM, I/O, average duration of garbage collection pauses, if applicable, and other metrics)

◊ Response time per application interface

◊ Excessive or insufficient logging

◊ Logging of sensitive information

◊ etc.

1.17 Summary

DevOps can be viewed as a cross-team software delivery discipline (paradigm)

The DevOps practice has been popularized by organizations adopting the Cloud-as-a-Service computing model

The following elements and capabilities need to be put in place for DevOps to meet its objectives:

◊ Alignment with the business needs

◊ Collaborative development

11


◊ Continuous testing and integration

◊ Continuous release and deployment

◊ Continuous application monitoring

12

Chapter 2 - Standing Up DevOps


DevOps Adoptions in the Enterprise

IT Governance

Select Techniques and Practices

Service Quality Metrics

2.1 Standing Up DevOps

There is no cast-in-stone rules on how to set up the DevOps capability

Every organization finds its own organizational form for DevOps

◊ Some companies create a joint DevOps group with a single reporting line

◊ Others establish a small contact group with representatives from all stakeholder groups

◊ Still others completely delegate Ops functions to Dev (mostly the case with Cloud-based shops)

2.2 Things to Look For and Avoid

Wrong people and inefficient processes may turn DevOps into a liability rather than an asset

Despite the "communal" nature of the DevOps operations, there should be a clear separation of operational scopes demarcated along discrete roles within DevOps; also, strict operational rules must be established and enforced, including:

◊ Production passwords must not be shared across DevOps

Failure to do so may result in developers sneaking in production environment with Ops' credentials to perform unauthorized actions

◊ All operational activities in production environment must be traceable with tamper-proof logging of such information as: user id, source IP


address, timestamp, and performed activity

◊ All admin scripts along with their supporting documentation must be checked in a version control system

◊ System and application production passwords must be kept in a safe way accessible only to the authorized personnel

2.3 IT Assets Ownership

What happens when something isn't “owned” by someone?

◊ “Tragedy of the commons” – Garrett Hardin, Science 1968

◊ “That which is common to the greatest number has the least care bestowed upon it.“ – Aristotle, Politics 1261b34

IT assets ownership is critical

◊ Accountability to the enterprise

◊ Evolution alongside changing business needs

◊ Motivation to maintain and support

◊ Quality service and customer satisfaction

Notes:

The Basic Idea of Garrett Hardin's article, “Tragedy of the commons”:

"If a resource is held in common for use by all, then ultimately that resource will be destroyed. "

"Held in common" means the resource is owned by no one, or owned by a group, all of whom have access to the resource.

2.4 Viewing Applications As Products, not Projects

In some cases, taking ownership of an IT asset can be promoted by adopting the software product model

The traditional model is project-centric where delivery of the project in production means the end of life of the project for the core development team who hands-off (and forgets) the deployed application to the maintenance team to support it

14


In some organizations, ownership of the application does not end there

◊ E.g. AWS uses this rule: "you build it, you run it", where a development team retains ownership of and responsibility for the application released in production. In this context, the application is treated as a product

Generally, the product development model (as opposed to the project model) leads to the feeling of "ownership" of the application (e.g. your team carries the application production pager 24/7 on a rotating basis), which contributes to the improved quality of the software product and yields other intangible benefits

2.5 DevOps in the Enterprise

While originally DevOps was popularized by Web (Cloud) -based companies, such as Flickr and Netflix, in one form or another, large enterprises have long been using DevOps practices

For deeper penetration of DevOps in the Enterprise space and establishing it as a true enterprise capability, it needs to be placed under control within the existing enterprise governance processes

Embracing DevOps practices help organizations to more efficiently and effectively manage aggressive software delivery schedules by minimizing the number of software defects and deployment failures

Also, in addition to the known benefits (higher delivery velocity and more predictable application development and production deployment cycles, etc.), DevOps help promote collaborative environment within organization

In some cases, a certain (sometimes very painful) shift in mentality and organizational culture is required to fully exercise the benefits of DevOps practice

2.6 IT Governance

IT governance is about managing IT resources / applications / systems and ensuring that IT decisions are aligned with strategic and operational business requirements

DevOps activities are no exception

15


IT governance processes are created with a view to

◊ Mitigate IT risks

◊ Make the outcome of IT activities predictable

◊ Measure IT performance

◊ Promote standards and best practices

◊ Establish proven policies and procedures to ensure projects' success

2.7 Governance and Risk Mitigation

In essence, governance is about mitigating risk

This risk mitigation is manifested in several ways

◊ Defined procedures, templates, and checklists

◊ Design-, Change-, Deployment- and Run-time management

◊ Enterprise-wide resources (technical and human)

◊ Documentation and promotion of reference architectures, design patterns, standards, and best practices

2.8 DevOps Adoption Steps

1. Identify your business drivers✔ Product time-to-market metric (faster time to value), etc.

2. Get educated✔ Learn / educate / evangelize about various DevOps techniques:

Continuous Integration, cross-team interests, transparency, etc.

3. Articulate DevOps' value proposition✔ Minimization of resource wasting (e.g. cutting down on avoidable

overtime), reduction of the number of defects, creation of the reliable software delivery pipe-line, etc.

16



4. Define one or more scenarios of software delivery with DevOps techniques✔ If possible, set up and run a [small] proof of concept (PoC) project

✔ Show how DevOps can address some of the business drivers

5. Produce a road map✔ Provide time-specific and quantifiable plans for installing DevOps-

related tools/systems, staff training, etc.

6. Gain stakeholder buy-in✔ Get all parties concerned on board with regard the DevOps initiative

✔ Keep them informed on the progress of the DevOps initiative

✔ Gently educate them, if needed


7. Establish governance for risk mitigation✔ This is particularly important for Enterprise environments

✔ Poorly managed DevOps activities can backfire on this initiative

✗ "Oops, we promoted the wrong code (there was no Change Request for that)"

8. Establish a core team✔ It may be a contact group with representatives from various

disciplines/departments

✔ Team members must be psychologically compatible -- some team-building events might help with identifying those candidates


9. Invest in infrastructure (not applicable if you operate in the Cloud)✔ Provision infrastructure for DevOps operations (servers, version control

17


system, etc.)

10. Pilot✔ You can build on any PoC projects you managed to sneak in Step 4

above

✔ Make sure you can demonstrate the repeatable and reliable nature of DevOps' software delivery process

11. Enterprise roll-out

✔ Don't forget that it was a joint effort

✔ Time to celebrate!

2.12 Select DevOps Techniques and Practices

The DevOps capability is supported by a number of common techniques and practices, including:

◊ Collaborative steering

◊ Continuous testing of all aspects of the application delivery pipe-line

◊ A Version Control System

◊ A Bug Tracking System

◊ Iterative and frequent integration and deployment

◊ Automation

◊ Change Management

◊ Monitoring and auditing


Collaborative steering ◊ Collaborative and transparent working environment promotes visibility

and agility of software development processes

◊ In order to receive timely notifications or feedback, you may want to set up a Web UI dashboard publishing application lifecycle status in real

18


time for all parties concerned

Continuous testing of all aspects of the application delivery pipe-line ◊ Where applicable, DevOps assures quality of software code;

deployment scripts for all environments (QA, UAT, Staging, etc.); scripts for setting up the infrastructure components (VMs, databases, application servers, etc.),


A Version Control System ◊ For change tracking and consistency of your application code, admin

and deployment scripts, keep them in a version control system (VCS). Changes in your VCS should be accompanied with a check-in message referencing the defect # or change request # addressed by the code check-in, e.g. cvs commit -m "Bug #12YYZ fix" stringUtils.c

A Bug Tracking System ◊ Defects and issues must be tracked, accounted for, and acted upon

◊ The use of a bug tracking system is regarded as a hallmark of a mature software engineering practice

Iterative and frequent integration and deployment ◊ With this practice in place, you can guarantee consistent and

predictable release times; applications can be installed reliably and repeatably


Automation◊ Processes that need manual intervention may occasional fail due to the

human factor (to err is human!)

◊ Establish the continuous automation practice. Keep automation scripts in your VCS

19


Change Management (CM) ◊ This is a critical aspect of DevOps that is often overlooked

◊ CM is a core part of the overall IT governance process

◊ CM helps track changes introduced to the target system by recording the reason for change, scope of change, references to the applicable VCS revisions of assets involved in the change, etc.

◊ CM as a system of record addresses the audit requirements and promotes transparency


Monitoring and auditing◊ Runtime parameters (response time, computing throughput, etc.) and

other metrics of the application (service availability, time to recover from an outage, etc.) deployed in production must meet the client's Service Level Agreement (SLA); the only practical way to capture those parameters is to run your application in a production-like environment

◊ Have a dedicated production-like Performance (a.k.a. Production) Testing Environment (PTE) to collect critical runtime parameters of the application before its release into production

Notes:

Cloud vendors offer run-time monitoring systems that provide sufficient data for clients to have a clear picture of their run-time environments. For example, AWS offers the CloudWatch monitoring system that lets developers view and monitor operational and performance metrics for AWS resources such as Amazon EC2 instances, EBS volumes, Elastic Load Balancers, and RDS DB instances. In addition, clients can add application metrics and business information to CloudWatch for monitoring alongside AWS resource metrics. Clients can use current and historical metrics to troubleshoot issues and discover trends, create and edit alarms to be notified of problems, etc.

20


AWS's CloudWatch Service Dashboard

2.17 Service Quality Metrics

An SLA lists contractual terms and mutual obligations of the client and the service provider and run-time metrics (Quality of Service parameters) to help physically measure SLA parameters, including:

◊ Availability – total up-time / total time per reporting period (month/year)

Up-time is affected by outages and interrupted service duration

Is also impacted by mean-time to enlist new resources and release of unneeded ones

◊ Reliability – guaranteed rate of successful responses

Expressed as mean-time between failures (MTBF)

◊ Performance – service delivery time guarantees

Network throughput, serviced requests per second, response time, etc.

◊ Resiliency – capability to efficiently absorb load spikes

21


2.18 Summary

There are no prescribed solutions for standing up the DevOps capability

In order to make it an integral part of the Enterprise, DevOps needs to be placed under control within the existing enterprise governance processes

Some of the techniques and practices that help with establishing DevOps capability in the Enterprise include:

◊ Collaborative steering

◊ Continuous testing of all aspects of the application delivery pipe-line

◊ A Version Control System

◊ Iterative and frequent integration and deployment

◊ Automation

◊ Change Management

◊ Monitoring and Auditing

22

Chapter 3 - DevOps Tools


Compare IaaS with PaaS

Overview of select DevOps tools

Overview of the IBM Collaborative Lifecycle Management unified platform

3.1 The Choice of Cloud Platform

The DevOps practice has been popularized by organizations adopting the Cloud-as-a-Service computing model (Netflix, Flickr, et al)

Many enterprises are adopting the Cloud-as-a-Service computing model either by setting up a Virtual Private Cloud (VPC) within a public cloud environment or setting up on-premise cloud-like environments

You have a choice between the IaaS and the PaaS platform options which have different implications for DevOps

3.2 IaaS for DevOps

An IaaS platform gives you the lowest level of access to cloud infrastructure: VMs, virtual networks and load balances, a choice of storage solutions (NoSQL, Relational Databases, etc.), etc.

◊ Popular IaaS platforms are: Amazon Web Services(AWS), Google Compute Engine, Microsoft Azure, Cloud Foundry, OpenStack, Rackspace

To effectively and efficiently manage large cloud environments, heavy DevOps involvement is needed

◊ DevOps will be responsible for patching OS / software, formatting raw block storage units, setting up security (open/close virtual firewall ports, managing ACL), etc.


3.3 PaaS for DevOps

A PaaS platform gives users a selection of sandboxed run-time environments, APIs for accessing managed storage and messaging systems, etc.

Popular PaaS platforms are: Microsoft Azure (built on top of the Azure IaaS platform), Google App Engine, AWS Elastic Beanstalk (built on top of the AWS IaaS platform), Heroku (acquired by Salesforce.com in 2010), OpenShift, CloudFoundry, IBM Bluemix (built on top of CloudFoundry)

Due to the managed nature of PaaS, most of the run-time and provisioning tasks are handled by the platform vendor; DevOps involvement is limited to code promotion and some allowed high-level environment tuning

In essence, DevOps perform the push, scale, update types of activities on PaaS

◊ The Ops side of DevOps can be safely handled by Dev alone

Notes:

The first release of the Bluemix Enterprise Cloud Platform (originally printed as BlueMix) was announced by IBM on 24 Feb 2014; the platform was positioned as

"… a unique new development environment and capabilities-as-a-service to help clients and developers speed the adoption of "hybrid" clouds, which have the potential to usher in a new era of innovation across the enterprise. As part of its initiative, IBM has invested more than $1 billion for software cloud development and is launching new capabilities running on SoftLayer." [http://www-03.ibm.com/press/us/en/pressrelease/43257.wss]

3.4 Containerization Tools

A popular approach to gain a better utilization of a single physical / virtual machine's resources is to use containerization tools that allow for creating and running multiple VMs in containers on a single control host

Popular containerization tools are:

◊ LXC

Works on Linux hosts by leveraging modern Linux kernel's cgroups

24


capability for resource containerization such that applications' view of the underlying OS is completely isolated

◊ Solaris Containers (Zones) Cause a very low CPU and RAM overhead

First bundled with Oracle Solaris 11 release

◊ Docker An open-source project that automates the deployment of

applications inside software containers (e.g. LXC)

3.5 System Configuration Automation and Management

For large environments, configuration automation and management becomes a dire necessity

Most of configuration automation and management systems are built around the "infrastructure-as-code" paradigm

Popular Configuration Management tools are:

◊ Puppet Open source tool written in Ruby; runs on Linux and Windows

Allows to manage system configuration declaratively via its domain-specific language (DSL)

You can get Enterprise level support from Puppet Labs, a privately held company behind Puppet

◊ Chef Used to manage both Linux and Microsoft Windows OSes

Written in Ruby and Erlang

Some claim that Chef is more flexible than Puppet, albeit at the expense of more complex system administration

3.6 System Configuration Automation and Management

SaltStack (or, simply, Salt)

25


◊ A Python-based open source configuration management system that competes primarily with Puppet, Chef, and Ansible

Ansible◊ A Python-based open source configuration management system

◊ Commercial support is provided by Ansible, Inc.

◊ Supports Red Hat, Debian, CentOS, OS X, and BSD Linux and Unix flavors; Windows OSes support started as of version 1.7

Ubuntu Juju◊ An open source service orchestration management tool sponsored by

Canonical, the company behind Ubuntu

◊ Juju provides services via charms ("smart" software bundles)

◊ Juju is used to install software, start/stop a service, manage relationships with other charms, upgrade charms, etc.

3.7 Continuous Integration (CI) Systems

Jenkins

◊ A Java-based open source continuous integration (CI) tool

◊ Forked from “Hudson” in 2010

Hudson is now part of Eclipse Foundation with much weaker traction in the CI IT community

◊ At its core, Jenkins is a Java Web server (e.g. Tomcat)

◊ It supports integration with a number of version control systems (VCS), including AccuRev, CVS, Git, Perforce, Subversion, Clearcase, et al

◊ Integrated with Apache Ant and Apache Maven build systems

TeamCity

◊ An automated build management system and CI server written in Java

◊ Sponsored by JetBrains (https://www.jetbrains.com/)

◊ TeamCity is a commercial product licensed from JetBrains

26


There is a free edition that supports up to 20 build configurations and 3 build agents

3.8 Build and Dependency Management Systems

Apache Ant◊ A Java-based command-line tool for automating software build

processes by way of scripted targets and tasks

◊ Does not impose any coding conventions nor prescribed directory layouts for build projects

◊ Uses Apache Ivy for dependency management

Maven◊ Distributed under Apache License 2.0

◊ Contrary to Apache Ant, Maven uses naming conventions and prescribed folder structure for the build processes

◊ Comes with many pre-defined targets for common project tasks (code compilation, packaging, etc.)

◊ Maven dynamically downloads the required dependencies (Java libraries and Maven plug-ins) from one or more Central repositories and stores them locally

3.9 Build and Dependency Management Systems

Gradle◊ A project automation tool that is built around concepts of Apache Ant

and Apache Maven

◊ Written in Java and Groovy

◊ Uses Groovy-based domain-specific language (DSL)

◊ Designed for managing large projects backed up by complex build graphs, optimizing the build time by skipping parts of the project which have already been built

27


◊ Gradle is tightly integrated with Ant importing Ant build files as external scripts to be executed

Apache Ivy

◊ A sub-project of the Apache Ant project for resolving project dependencies

◊ To some extend, Ivy competes with Apache Maven, which also manages dependencies

However, Maven does more: it is a complete build system

3.10 Select DevOps Tools

Issues & Project Management Software◊ JIRA, Bugzilla, Trello

Monitoring, Alerting, and Trending:◊ Cacti, Ganglia, Graphite, Icinga, Nagios, New Relic, PagerDuty

Logging◊ Loggly, Logstash, PaperTrail, Splunk, SumoLogic

Process Supervisors◊ Bluepill, Monit , Upstart, systemd

3.11 Collaborative Lifecycle Management Solutions from IBM

IBM identifies Collaborative Lifecycle Management (CLM) as a holistic discipline focusing on improving software quality and increasing the velocity of software delivery

The CLM capability is built around a combination of Enterprise practices and disciplines, including:

◊ requirements management

◊ quality management

◊ change and configuration management

28


◊ project planning and tracking

3.12 The Collaborative Lifecycle Management Diagram

Collaborative Lifecycle Management connects business analysts, developers, and testers

Source: IBM Knowledge Center

3.13 The IBM Collaborative Lifecycle Management Platform

IBM integrates a number of software delivery tools and systems into a unified Collaborative Lifecycle Management (CLM) platform:

◊ Rational Team Concert

◊ Rational Quality Manager

◊ Rational DOORS Next Generation

The CLM platform brings together the complete set of application lifecycle management (ALM) capabilities that are mandated in the Enterprise space

3.14 Rational Team Concert (RTC)

Rational Team Concert (RTC) creates one single agile project

29


collaborative environment for:

◊ Task and issue tracking

◊ Source control

◊ Agile project management and planning

◊ Continuous Builds and Integration

Some DevOps staff working with RTC mention its high price and not consistent quality across the functional areas in its first versions

RTC is free for teams up to 10 developers

Notes:

RTC is shipped with adapters for HP ALM, Atlassian JIRA and open source Git should you wish to integrate with your existing Application Lifecyle Management systems.

Source: IBM RTC

3.15 Rational Quality Manager (RQM)

Rational Quality Manager (RQM) is positioned as a "collaborative hub" for the Quality Assurance aspect of Application Lifecyle Management (ALM)

RQM has the following capabilities:

◊ Test planning and management

◊ Test design, creation, and execution

30


◊ Traceability with requirements

◊ Change management

RQM provides public REST API for CRUD (create, read, update and delete) operations against its repositories

3.16 Rational DOORS Next Generation (DNG)

Rational DOORS Next Generation (DNG) is a requirements management solution

It helps with activities related to requirements capturing, storing, tracing and management

◊ DNG helps assess the impact of any planned application change by linking submitted change requests to original requirements

DNG provides a secure repository for storing requirements as a Word document or an Excel spreadsheet (including CSV files)

DNG help maintain compliance with industry standards and applicable regulations

It integrates with IBM Rational Team Concert, IBM Rational Quality Manager and IBM Rational Rhapsody Design Manager on IBM Jazz collaborative lifecycle management platform

3.17 Summary

To effectively and efficiently manage large environments on an IaaS Cloud platform, heavy DevOps involvement is needed

DevOps involvement in a PaaS environment is limited to simple code promotion activities and some high-level environment tuning

DevOps practice has a wide variety of tools for the job at its disposal from both open source and closed source projects

IBM is very active in the area of Collaborative Lifecycle Management (CLM) offering clients the unified CLM platform, which is built around the following systems:

31


◊ Rational Team Concert

◊ Rational Quality Manager

◊ Rational DOORS Next Generation

32

Chapter 4 - Containerization Systems Overview


Overview of virtualization

Overview of containerization

Compare virtualization with containerization

Review areas of effective use of virtualization and containerization

4.1 Virtualization

Virtualization is a technique of abstracting computer resources (hardware and networking) and allowing software (OSes and applications) to run in such environments as if it were a real computer

The collection of the virtualized resources are packaged in a Virtual Machine

Virtual Machines are managed by hypervisors (Virtual Machine Managers)

Virtualization helps drive server consolidation initiatives that leads to better resource utilization, management and driving operational costs down in public cloud computing, on-premise private clouds, and traditional enterprise data centers

Virtualization was fist introduced in mainframes back in 70s

4.2 Hypervisors

A hypervisor (or Virtual Machine Monitor (VMM)) is a specialized software program that creates and runs virtual machines

Hypervisors allow several operating systems (called "Guest OSes") to share a single hardware host at the same time in a way that

◊ Each guest OS receives its own fair share of virtualized resources (CPU, RAM, network, file-system, etc.)

◊ Guest OSes running on the same host do not affect others (allowing for


a safe multi-tenant hosting arrangement)

4.3 Hypervisor Types

There are two types of hypervisors:

◊ Type 1 (native) hypervisors

◊ Type 2 (hosted) hypervisors

4.4 Type 1 hypervisors

Type 1 (native) hypervisors run directly on the host's hardware and manage guest OSes that are executed just a lever above the hypervisor

◊ Examples of Type 1 hypervisors: the Citrix XenServer, VMware ESX/ESXi, KVM, Microsoft Hyper-V, Oracle VM Server for SPARC, Oracle VM Server for x86 hypervisor

4.5 Type 2 hypervisors

Type 2 (hosted) hypervisors run as a system program on a regular operating system environment (FreeBSD, Linux, or Windows.). The guest OSes run on top of the hypervisor

34


◊ Examples of Type 2 hypervisors: VMware Workstation and VirtualBox are examples of Type 2 hypervisors

4.6 Type 1 vs Type 2 Processing

Processing occurs differently between these two basic types of VM. Type 1 relies upon native hardware that supports virtualization whereas the Type 2 introduces a software abstraction layer to facilitate the virtualization

35


4.7 Paravirtualization

Paravirtualization (PV) is a special case of system virtualization

With PV, a hardware environment is not entirely simulated and the guest OSes get a coordinated access to the underlying physical hardware bypassing the VMM in certain cases

PV requires the guest OSes to be modified (ported for the PV-API )

A popular PV hypervisor Xen is developed under the Xen project using the GNU GPL v2 license (http://www.xenproject.org/)

PV hypervisors have a simple design and offer excellent execution performance

◊ For example, with Xen you get performance degradation between 2% for a standard workload and 10 – 20% for a worst-case scenario

4.8 Virtualization Qualities (1/2)

Transparent sharing of resources (memory, network, disk, CPU, etc.)

◊ decouples hardware from software

◊ hardware aggregates as a pool of sharable resources

Live migration

◊ shift virtual servers between physical hardware instances while running

◊ facilitate zero downtime while still maintaining hardware

Isolation

◊ limits security exposure

◊ reduces spread of risk

4.9 Virtualization Qualities (2/2)

Management

◊ single point of control across all VMs

◊ ease deployment burden through repetitive scripts and templates

36


◊ block-level rollbacks to prior snapshots in the event of failure

High availability

◊ boot virtual servers on alternate hardware in the event of primary hardware failure

◊ execute multiple instances of a virtual server across multiple hosts

4.10 Disadvantages of Virtualization

System virtualization tools or emulators need to load and boot up a full image of the guest OS and, basically, need to emulate a complete machine, which results in a high operational overhead

The virtualization overhead sharply increases in proportion to the number of guest OSes running on the same physical host:

◊ CPU (CPU caches efficiencies fall)

◊ Memory (swapping increases)

◊ IO / Networking (contention increases)

◊ The underlying scheduler contention

4.11 Containerization

Containerization is a lightweight alternative to full machine virtualization

◊ It is often called virtualization environment, or OS-level virtualization

A container encapsulates an application running inside its own operating environment which is derived from the underlying host OS

Containerization has been popularized by cloud-like processing environments that require fast server boot-up time

At the moment, the most widely used technology behind containerization is LinuX Containers (LXC), which is a userspace interface for the Linux kernel containment features (cgroups and namespaces)

The main containerization action is happening in the Linux space (and x84 (32-bit) & x64 (64-bit) machine architecture)

37


4.12 Virtualization vs Containerization

Both offer multi-tenancy for guests (OSes and applications)

Virtualization is about translating communication between the hosted OSes and hypervisor

Containerization is "native" in that containers share the host OS's kernel

◊ Containers' OS is the same as the hosts' OS

Virtualization allows to run multiple guest OSes on the same host, while containerization is limited to the OS type the host uses

Traditional virtualization offers better protection from "rogue" tenants

4.13 Where to Use Virtualization and Containerization

Virtualization belongs to the Enterprise space (and big Ops budgets)

If you an infrastructure provider and a Linux shop competing on price, use containerization

◊ Platform-as-a-Service (PaaS) vendors such as Heroku, OpenShift, and Cloud Foundry use Linux containerization

Containerization is the way to go if you are cost-conscious organization trying to save every nickel (most Linux containerization systems are free)

DevOps can hugely benefit from containerization as well (fast system provisioning: build, test, inergration machines, etc.)

4.14 Popular Containerization Systems

Linux containerization systems:

◊ LXC (more on LXC a bit later ...)

◊ Docker (more on Docker a bit later ...)

◊ OpenVZ (more on OpenVZ a bit later ...)

◊ Linux-VServer Non-Linux systems containerization systems:

38


◊ Oracle Solaris Zones (Containers) (more on them a bit later ...)

◊ IBM AIX Workload Partitions◊ FreeBSD Jails

4.15 What are Linux Containers

LinuX Containers (LXC) is an OS-level virtualization environment that allows multiple Linux systems to run on a single physical host

LXC provides a user API and a toolset for interfacing with the Linux kernel containment features (cgroups and namespaces)

The main differentiating factor between LXC and some other systems is that it runs on the unmodified Linux kernel of various Linux distros

The LXC system is written in a number of programming languages, including C, Python 3, Lua, and others

Initial release was in August 2008

4.16 Docker

Docker is an open-source system for creating virtual environments

Runs on any modern-kernel Linux distributions

To communicate with the underlying host kernel, Docker uses virtualization interfaces based on a variety of systems:

◊ libvirt

◊ LXC (LinuX Containers)

◊ systemd-nspawn

You can install Docker inside a VirtualBox and run it on OS X or Windows

Docker can be booted from the small footprint Linux distribution boot2docker

Written in the Go programming language

39


4.17 OpenVZ

OpenVZ is a Linux OS-level virtualization technology that leverages Linux modern features kernel

OpenVZ is similar to FreeBSD jails and Oracle Solaris Zones

Requires Linux kernel patching

◊ As of version 4.0, OpenVZ can work with unpatched Linux 3.x kernels offering a reduced functionality

Written in C

4.18 Solaris Zones (Containers)

Oracle Solaris Zones (previously known as Solaris Containers) is an OS-level virtualization technology for Interl-based and SPARC systems

Oracle Solaris Zones are an integral part of the Oracle Solaris 10 and up

Oracle promotes this technology as a way to maintain the one-application-per-server deployment model that share same hardware resources

The first public release was in February 2004

4.19 Summary

System virtualization tools or emulators offer great tenant (guest OSes) isolation at the expense of relatively high operational overhead

Traditional virtualization is suitable for Enterprise use cases

Containerization is an OS-level lightweight alternative to traditional virtualization offering uses the following benefits:

◊ Better runtime performance and system boot-up times

◊ Overall cheaper solutions (most containerization systems and tools are free)

◊ Containers is an excellent choice if you are an infrastructure service provider of Linux-based systems

Containerization has some disadvantages:

40


◊ Relatively low protection against "rogue" tenants

◊ Containers launched on a host will inherit the host's OS, which limits your choices

◊ Currently very popular only in the Linux world

41

ùcu Ç|& ëÊ· n o=ÔûbüÜxyÎ ÈÄi¸ g1 k · 2017-11-06 · 1.8 infrastructure as code...

Documents