session 3 - cloudstack test automation and ci

2013 Trend Micro

25th Anniversary

TCloud Computing, Inc.

CloudStack

test automation and CI

2013 Trend Micro

25th Anniversary

Agenda

CloudStack Test Automation

– Deploy and Test, Testing Framework Marvin

CloudStack Continuous Integration by Jenkins

– High Level Architecture

– Build Process

– Deploy Process

– Test Process

2013 Trend Micro

25th Anniversary

CloudStack Test Automation - Marvin

2013 Trend Micro

25th Anniversary

What to test Apache CloudStack

What to Test and Verify

• RESTful API

• Virtual Machine

• Hypervisors

• Storage Pool

• Virtual Network

• Database

Management Server

MySQL

DB

Cloud Infrastructure (Hypervisors and

Storages)

User API

Admin API

RESTful API User Interface

Virtual Machine Database

Hypervisors and Storages

2013 Trend Micro

25th Anniversary

How to test Apache CloudStack

How to Test and Verify ?

(Manual Test)

• Web Browser

• XenCenter

• SSH client tools

• mysql client tools

Cloud we speed up the testing?

Management Server

MySQL

DB


Storages)

User API

Admin API

Web Browser SSH Client MySQL Client

XenCenter

SSH Client

2013 Trend Micro

25th Anniversary

CloudStack project - Marvin

Automation Test

Log Deployment

2013 Trend Micro

25th Anniversary

CloudStack Test Automation - marvin

Apache CloudStack Test Automation – marvin

• testing framework, Python 2.7, unittest2, paramiko, mysql …

Marvin Source:

• <CloudStack Source>/tools/marvin

Marvin features to verify:

• Binding to CloudStack API (synchronous and asynchronous job)

• Support Deployment and Testing

• Support Log and Report

• Support SSH Client

• Support Database Client

2013 Trend Micro

25th Anniversary

CloudStack Test Automation – marvin structure

Marvin

Test Client

Management Server MySQL

DB


Storages)

API Client Verify

Database Client Verify

SSH Client Verify

2013 Trend Micro

25th Anniversary

CloudStack Test Automation – marvin process

Marvin

Test Client

Load Configuration (datacenterCfg)

Run Test Suite

Execute Deployment

Generate logs and reports

1.Load JSON format

- define cloud infrastructure

- define database location

- define Management Server

2. Create api/secret key

1. Deploy Cloud Infrastructure

by cloudstack API automatically

2. Include Zone, Pod, Cluster,

Host, Storage, Network

1.Load unittest2 module and the

Test Suite you define to verify.

2. Can discover all Test folder

3. Start to Test!

1. Create Test Client log for all

the execution.

2. Create Test Case Result

2013 Trend Micro

25th Anniversary

CloudStack Test Automation – marvin

Test Automation marvin requirement:

• CentOS 6.3/Ubuntu

• Python 2.7

• python-pip

cd <CloudStack_Source>

To build marvin: (Be sure you can build CS first)

- ant build-marvin

To packe marvin:

- ant package-marvin

To install marvin

- ant install-marvin

2013 Trend Micro

25th Anniversary


Step 1

Configure datacenterCfg in JSON format

• Zone

• Pod

• Cluster

• Host

• Primary Storage

• Secondary Storage

• Network Information

• Database information

• Management Server Information

datacenterCfg {

"zones": [

{

"name": "zone01",

"guestcidraddress": "10.1.1.0/24",

"dns2": "8.8.8.8",

"dns1": “4.4.4.4",

...

},

"dbSvr": {

"dbSvr": "10.1.191.251",

"passwd": "cloud",

"db": "cloud",

"port": 3306,

"user": "cloud"

},

"mgtSvr": [

{

"mgtSvrIp": "10.1.191.251",

"port": 8096

}

]

}

2013 Trend Micro

25th Anniversary


Step 2

Write Your Own Test Case by unittest2

== test_SampleCode.py ==

class Test_SampleCode(cloudstackTestCase):

def setUp(self):

self.apiclient = self.testClient.getApiClient()

def tearDown(self):

pass

def test_apiclient(self)

listZones_parms = listZones.listZonesCmd()

res = self.apiClient.listZones(listZones_parms)

if not res.id:

self.fail('zone is empty')

# cat /tmp/results.log

test_apiclient (Test_SampleCode.Test_SampleCode) ... Ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK

2013 Trend Micro

25th Anniversary


Three Types of testClient (apiClient/sshClient/dbClient)

• CloudStack API client

listZones_parms = listZones.listZonesCmd()

res = self.apiClient.listZones(listZones_parms)

zone_id= self.testClient.executeSql('select id from datacenter')

sshconnect = remoteSSHClient('10.1.191.251', 22, 'root', ‘password')

sshconnect.execute('ls')

• SSH client

• Database client

2013 Trend Micro

25th Anniversary


The debug method for log

• debug method.

sysvms = listSystemVms.listSystemVmsCmd()

sysvms.zoneid = z.id

self.debug("looking for system VMs in zone: %s, %s"%(z.id, z.name))

2013-03-01 17:15:08,394 - DEBUG - testclient.testcase.TestSetupSuccess - looking for

system VMs in zone: cbf16757-eb92-43f2-bb4b-8a46f3955bf9, zone01

2013-03-01 17:15:08,394 - DEBUG - testclient.testcase.TestSetupSuccess - looking for

system VMs in zone: cbf16757-eb92-43f2-bb4b-8a46f3955bf9, zone01

• debug messages.

2013 Trend Micro

25th Anniversary


Step 3:

To run the Test

# cd <CloudStack_Source>/tools/marvin/marvin

Deployment and Automation test: # python deployAndRun -c datacenterCfg -t /tmp/testcase.log -r /tmp/results.log -f sandbox/testSetupSuccess.py

option purpose

-c points to the configuration file defining our deployment

-r test results log where the summary report is written to

-t testcase log where all the logs we wrote in our tests is output for debugging purposes

-d directory containing all the test suites

-l only load the configuration, do not deploy the environment

-f Run tests in the given file

2013 Trend Micro

25th Anniversary


Step 4:

Check the Testing Result:

test_01_create_disk_offering (test_disk_offerings.TestCreateDiskOffering)

Test to create disk offering ... ok

test_02_edit_disk_offering (test_disk_offerings.TestDiskOfferings)

Test to update existing disk offering ... ok

test_03_delete_disk_offering (test_disk_offerings.TestDiskOfferings)

Test to delete disk offering ... ok

test_deploy_vm (test_vm_life_cycle.TestDeployVM)

Test Deploy Virtual Machine ... Ok

----------------------------------------------------------------------

Ran 4 tests in 127.529s

2013 Trend Micro

25th Anniversary

CloudStack provides ..

<CloudStack_Source>\test\integration

• Smoke Test

• Integration Test

2013 Trend Micro

25th Anniversary

Summary - Marvin

It can help you:

• Automate and speed up the test

• Fast deployment for Zone/Pod/Cluster/Host

• Generate the test result and test log

Development purpose:

• Automation Test

• Continuous Integration

IT and operation purpose:

• Automated operation

• Proof of concept

• Fast deployment

2013 Trend Micro

25th Anniversary

Cloudstack Continuous Integration by jenkins

2013 Trend Micro

25th Anniversary

CloudStack on Continuous Integration

Advantages:

• Automate the build

– daily and every commit

• Automate the deployment and self-testing

• Keep the build fast and stable

• Release Purpose

CloudStack Continuous Integration Process

• Build Process

• Deploy Process

• Test Process

• Release Process

2013 Trend Micro

25th Anniversary

Jenkins

Simple Continuous Integration flow

Source Version Control

(GIT/SVN..)

Job Get Source

Get, Build, Run and Get ...

Job Job

Job Job Job

Trigger the process by

• Schedule

• Poll SCM

2013 Trend Micro

25th Anniversary

Master/Slave Architecture

Jenkins Slave

Job Job Job

Jenkins

Master

Job Job Job

Jenkins Slave

Job Job Job

Job Job Job

Subversion Version Control

(GIT/SVN..)

Allocate Jobs

Poll SCM

Jenkins Slave

Job Job Job

Job Job Job

2013 Trend Micro

25th Anniversary

CloudStack on Jenkins CI: Term and Roles

Jenkins Roles:

• Master: Jenkins Manager

• Slave: Execute Jenkins Agent, It can execute jobs of the project

The Roles of CloudStack

• Jenkins Master: The Central Manager.

• Jenkins Slave: CloudStack Management Servers

• Hypervisors: XenServers

• Storage: Primary/Secondary Storages

• Test Clients: marvin

2013 Trend Micro

25th Anniversary

CloudStack on Jenkins CI

Jenkins Master

(Jenkins Slave) CloudStack

Management Server

Deployment

Module


Cloud Infrastructure

XenServer XenServer

XenServer XenServer

XenServer XenServer

XenServer Storage

3. Trigger Deployment

Test Client Test Client

1. Allocate Jobs

2. Get Source and Build

4. Deploy Cloud Infra.

5. Run Test Clients

…

2013 Trend Micro

25th Anniversary

CloudStack Continuous Integration

CloudStack Continuous Integration Cycle:

Build

Process

Deploy

Process

Test

Process

Release

Process

2013 Trend Micro

25th Anniversary

Cloudstack Continuous Integration

Build Process

• Retrieve the source code

• System environment setup

• Development environment setup

• Build and deploy to apache-tomcat

• Start the CloudStack Service

Jenkins Master


Management Server


Build Process

Deploy Process

Test Process

Release Process

2013 Trend Micro

25th Anniversary

Cloudstack Continuous Integration

Deploy Process

• Deployment Module:

– cobbler/puppy:

• Install/Configure Host/Storages

– Hypervisors on Virtual Machine (nested VM)

• Take Snapshot and Rollback for each run

• CloudStack deployment:

– Use testing framework marvin to deploy CS

– Zone/Pod/Cluster/Host/Hypervisors/Storages

– Make sure SystemVMs start correctly

Deployment

Module


XenServer XenServer

XenServer XenServer

XenServer XenServer

XenServer Storage

2013 Trend Micro

25th Anniversary

Cloudstack continuous integration

Test Process

• Run Test Suite

• Testing Framework marvin

– Test and verify cloudstack API

– Test by SSH Client

– Test by Database Client

• Generate Test Results and Reports


Management Server

Test Client Test Client


…

2013 Trend Micro

25th Anniversary

Cloudstack continuous integration

Release Process

• Integrate the Documents and Reports

• Run the release procedure to build install package

• Notification.

Build

Process

Deploy

Process

Test

Process

Release

Process

2013 Trend Micro

25th Anniversary

2013 Trend Micro

25th Anniversary

Q & A

session 3 - cloudstack test automation and ci

Technology

apiclient test

test suite

class test

test folder

test client log

process test process

trend micro

test case result