01

About me02

Andrey AdamovichBio: Developer, coach, speaker, author

Company: Aestas/IT (http://aestasit.com)

E-mail: andrey@aestasit.com

Linkedin: http://www.linkedin.com/in/andreyadamovich

Twitter: @aestasit

•••••

03

What's this presentation about?Our take on:

DevOps

Intrastructure Provisioning

Continuous Integration

Continuous Delivery

••••

04

TechnologiesGroovy - http://groovy.codehaus.org

Gradle - http://gradle.org

Jenkins - http://jenkins-ci.org

Puppet - http://puppetlabs.com

AWS - http://aws.amazon.com

•••••

05

Developers +Operations =

?06

Silos

07

Conflicts08

Risk

09

Agile

10

What is DevOps?

11

C.A.M.S.Culture : People over processes and tools. Software is made by and

for people.

Automation : Automation is essential for DevOps to gain quick

feedback.

Measurement : DevOps finds a specific path to measurement. Quality

and shared (or at least aligned) incentives are critical.

Sharing : Creates a culture where people share ideas, processes, and

tools.

•

•

•

•

12

It's not abouttools!

13

It's aboutculture!

14

We startedwith the tools!

15

Infrastructure as codeAutomate the provisioning and maintenance of servers:

Build from source control

Utilize existing tools

Ensure testability

•••

16

Configuration propagation

17

Configuration propagation

18

ChangesImagine manually uploading *.class files and repackaging JAR

directly on production servers when you have an urgent code change.

19

Deploymentis automatic!

20

And, so,should be...

21

infrastructureconfiguration

changes!22

No manualchanges!

23

Building an automation toolkitAutomation is key

We are JVM hackers

Fragmented ecosystem

•••

24

Initial toolsetGradle

Groovy

Ant

Python/WLST

Shell scripts

•••••

25

Required toolingInfrastructure connectivity

Infrastructure provisioning

Infrastructure virtualization

Infrastructure testing

••••

26

First Blood27

Ant + Gradleant.taskdef(

name: 'scp',

classname: 'o.a.t.a.t.o.ssh.Scp',

classpath: configurations.secureShell.asPath)

ant.taskdef(

name: 'sshexec',

classname: 'o.a.t.a.t.o.ssh.SSHExec',

classpath: configurations.secureShell.asPath)

01.

02.

03.

04.

05.06.

07.

08.

09.

28

Simple callant.sshexec(

host: host,

username: user,

password: password,

command: command,

trust: 'true',

failonerror: failOnError)

01.

02.

03.

04.

05.

06.

07.

29

Next step: wrapper functiondef ssh(String command,

Properties props,

boolean failOnError = false,

String suCommandQuoteChar = "'",

String outputProperty = null) {

...

}

01.

02.

03.

04.

05.

06.

07.

30

Next step: wrapper functiondef scp(String file,

String remoteDir,

Properties props) {

...

}

01.

02.

03.

04.

05.

31

Task example Itask installFonts << {

forAllServers { props ->

ssh('yes | yum install *font*', props)

}

}

01.

02.

03.

04.

05.

32

Task example IItask uninstallNginx << {

forAllServers { props ->

ssh('/etc/init.d/nginx stop', props)

ssh('yes | yum remove nginx', props, true)

ssh('rm -rf /etc/yum.repos.d/nginx.repo', props)

ssh('rm -rf /var/log/nginx', props)

ssh('rm -rf /etc/nginx /var/nginx', props)

}

}

01.

02.

03.

04.

05.

06.

07.

08.

09. 33

DrawbacksNew connection each time

Excplicit repeating parameters

Complex scripts are hard to maintain

Tasks are not idempotent

••••

34

Sshoogr

35

Sshoogr featuresGroovy-based SSH DSL for:

Remote command execution

File uploading/downloading

Tunneling

•••

36

Why Groovy?Groovy is perfect choice for scripting

Gradle build scripts are Groovy

Very mature, concise syntax

Extremely easy to produce DSL

We wrote a book about it!

•••••

37

Shameless plug

38

Sshoogr usage (import)@Grab(

group='com.aestasit.infrastructure.sshoogr',

module='sshoogr',

version='0.9.16')

import static com.aestasit.ssh.DefaultSsh.*

01.

02.

03.

04.

05.

39

Sshoogr usage (defaults)defaultUser = 'root'

defaultKeyFile = new File('secret.pem')

execOptions {

verbose = true

showCommand = true

}

01.

02.

03.

04.

05.

06.

40

Sshoogr usage (connection)remoteSession {

url = 'user2:654321@localhost:2222'

exec 'rm -rf /tmp/*'

exec 'touch /var/lock/my.pid'

remoteFile('/var/my.conf').text = "enabled=true"

}

01.

02.

03.

04.

05.

06.

41

Sshoogr usage (multi-line content)remoteFile('/etc/yum.repos.d/puppet.repo').text = '''

[puppet]

name=Puppet Labs Packages

baseurl=http://yum.puppetlabs.com/el/

enabled=0

gpgcheck=0

'''

01.

02.

03.

04.

05.

06.

07.

42

Sshoogr usage (file copying)remoteSession {

scp {

from { localDir "$buildDir/application" }

into { remoteDir '/var/bea/domain/application' }

}

}

01.

02.

03.

04.

05.

06.

43

Sshoogr usage (command result)def result = exec(command: '/usr/bin/mycmd',

failOnError: false, showOutput: false)

if (result.exitStatus == 1) {

result.output.eachLine { line ->

if (line.contains('WARNING')) {

throw new RuntimeException("Warning!!!")

}

}

}

01.

02.

03.

04.

05.

06.

07.

08.

09. 44

Sshoogr usage (shortcuts)if (ok('/usr/bin/mycmd')) {

...

}

if (fail('/usr/bin/othercmd')) {

...

}

01.

02.

03.

04.

05.

06.

45

Sshoogr usage (tunnels)tunnel('1.2.3.4', 8080) { int localPort ->

def url = "http://localhost:${localPort}/flushCache"

def result = new URL(url).text

if (result == 'OK') {

println "Cache is flushed!"

} else {

throw new RuntimeException(result)

}

}

01.

02.

03.

04.

05.

06.

07.

08.

09. 46

Sshoogr usage (prefix/suffix)prefix('sudo ') {

exec 'rm -rf /var/log/abc.log'

exec 'service abc restart'

}

suffix(' >> output.log') {

exec 'yum -y install nginx'

exec 'yum -y install mc'

exec 'yum -y install links'

}

01.

02.

03.

04.

05.

06.

07.

08.

09. 47

Still problemsComplex scripts are still not easy to maintain

Scripts are usually not idempotent••

48

Puppet49

Why Puppet?More mature than competition

Large community

Readable DSL

Good acceptance from DEVs and OPs

No need to learn Ruby ;)

•••••

50

Puppet example

51

Puppet provisioning

52

Puppet provisioning

53

Puppet provisioning

54

Puppet provisioning

55

Puppet state management

56

Puppet state management

57

Puppet state management

58

Puppet modules

59

Puppet modules

60

Puppet modules

61

Sshoogr +Gradle +Puppet62

Upload modulestask uploadModules << {

remoteSession {

exec 'rm -rf /tmp/repo.zip'

scp {

from { localFile "${buildDir}/repo.zip" }

into { remoteDir "/root" }

}

...

01.

02.

03.

04.

05.

06.

07.

08.

63

Upload modules ...

exec 'rm -rf /etc/puppet/modules'

exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'

}

}

01.

02.

03.

04.

05.

64

Apply manifeststask puppetApply(dependsOn: uploadModules) << {

remoteSession {

scp {

from { localFile "${buildDir}/setup.pp" }

into { remoteDir "/tmp" }

}

exec 'puppet apply /tmp/setup.pp'

}

}

01.

02.

03.

04.

05.

06.

07.

08.

09. 65

What we solved?Separated infrastructure state description and operations tasks

Scripts became more maintainable and idempotent••

66

In the meanwhile...We started developing complex/generic Puppet modules

Modules need proper testing

...on different platforms

•••

67

Do you test, right?How to test this stuff?

How to reuse a JUnit approach to testing?

We wanted things to be SIMPLE!

•••

68

PUnit

69

PUnitSimple testing tool for verifying remote server state

Uses Sshoogr and JUnit

Reuse reporting features of JUnit

As simple as ...

••••

70

PUnit example (derby)class DerbyInstallTest

extends BasePuppetIntegrationTest {

@Before

void installDerby() {

apply("include derby")

}

...

}

01.

02.

03.

04.

05.

06.

07.

08.

71

PUnit example (derby)@Test

void ensureDerbyRunning() {

command('service derby status > derbystatus.log')

assertTrue fileText("/root/derbystatus.log")

.contains('Derby')

assertTrue fileText("/root/derbystatus.log")

.contains('is running.')

}

01.

02.

03.

04.

05.

06.

07.

08.

72

PUnit example (derby)@Test

void ensureCanConnect() {

Thread.sleep(10000)

uploadScript()

command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' +

'testDataScript.sql > derbytest.log')

...

01.

02.

03.

04.

05.

06.

07.

73

Continuous integration

74

JenkinsDe-facto standard

Stable

There is a plugin for that!

•••

75

Jenkins build

76

Nextproblem?

77

ScalabilityHow do we test on different OS?

How do we run parallel tests on multiple architectures?

How do we avoid selling our houses?

•••

78

Amazon WebServices

79

Elastic Compute CloudMature

Great API

Virtual hardware variety

OS variety

••••

80

Gramazon

81

GramazonGroovy-based API for interacting with EC2

Integration with Gradle••

82

Gramazon example Itask startInstance(type: StartInstance) {

keyName 'cloud-do'

securityGroup 'cloud-do'

instanceName 'gramazon/cloud-do'

stateFileName 'cloud-do.json'

ami 'ami-6f07e418'

instanceType 't1.micro'

waitForStart true

}

01.

02.

03.

04.

05.

06.

07.

08.

09. 83

Gramazon example IItask terminateInstance(type: TerminateInstance) {

stateFileName 'cloud-do.json'

}

01.

02.

03.

84

The flowStart instance(s)

Upload manifests

Run tests

Generate report

Terminate instance(s)

1.

2.

3.

4.

5.

85

Next issue?86

Imgr

87

ImgrA tool for building images

Inspired by Packer••

88

SupportsShell

Puppet••

89

Configuration example

90

Summary91

Images, manifests, tasks

92

The big picture

93

Aetomation

94

ConclusionsReuse your existing Java knowledge

...to build a bridge between DEVs and OPs

Reuse development best practices for OPs

Don't be afraid to try new technologies

Infrastructure configuration is outsourcable now!

Automate!

••••••

95

Next steps?Create more documentation and examples

Add more DSL convience methods

Extend integration with Gradle

Add Windows connectivity/scripting support (groowin)

Define richer model for EC2 and potentially other clouds

Extend support for other provisioning tools

••••••

96

Readingmaterial

97

The Phoenix Project

98

Continuous Delivery

99

Release It

100

Programming Amazon EC2

101

Gradle in Action

102

Groovy 2 Cookbook

103

Technologies to followVagrant - http://www.vagrantup.com/

Docker - https://www.docker.io/

Packer - http://www.packer.io/

Qemu - http://wiki.qemu.org/

jclouds - http://jclouds.apache.org/

Cloudbees - http://www.cloudbees.com/

••••••

104

One morething...

105

It's all OpenSource!

106

Source codeSshoogr: https://github.com/aestasit/sshoogr

Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle

PUnit: https://github.com/aestasit/puppet-unit

Gramazon: https://github.com/aestasit/gramazon

Imgr: https://github.com/aestasit/imgr

•••••

107

Seekingcontributors!

108

Questions?109

Thank you!

110