staf 在自動化測試上的延伸應用 -- tmstaf (trendmicro staf)

李志和 (Bevis Lee)Sr. Engineer, R&D Dept.

2012/06/10

STAF 在自動化測試上的延伸應用 –TMSTAF (TrendMicro STAF)

6/12/20121 | Copyright 2012 Trend Micro Inc.

Agenda

• Brief Introduction of STAF

• What‟s TMSTAF?

• TMSTAF vs. QA automation

• Q&A

6/12/2012 2| Copyright 2012 Trend Micro Inc.

Brief Introduction of STAF

6/12/2012 3| Copyright 2012 Trend Micro Inc.

• History

– Created at IBM Austin in 1998

– Open-sourced (on SourceForge) in 2001

• http://staf.sourceforge.net/

46/12/2012 | Copyright 2012 Trend Micro Inc.

• STAF runs as a daemon process, called STAFProc

6/12/2012 5| Copyright 2012 Trend Micro Inc.

What‟s TMSTAF?

6/12/2012 6| Copyright 2012 Trend Micro Inc.

6/12/2012 7| Copyright 2012 Trend Micro Inc.

TMSTAF = Trend Micro STAF

6/12/2012 8Confidential | Copyright 2012 Trend Micro Inc.

TMSTAF Architecture

6/12/2012 9| Copyright 2012 Trend Micro Inc.

File Structure

6/12/2012 10| Copyright 2012 Trend Micro Inc.

6/12/2012 11Confidential | Copyright 2012 Trend Micro Inc.

TMSTAF Features

6/12/2012 12| Copyright 2012 Trend Micro Inc.

Testware configurations can be customized

Test suite, test case defined in Python syntax

Cross-platform execution

Command line interface

Testware concept included:

Test process included: Setup, Test and Teardown

Execution log for tracing

XML test report w/ email notification

Testing Report

6/12/2012 13| Copyright 2012 Trend Micro Inc.

Email Notification of Testing Result

6/12/2012 14| Copyright 2012 Trend Micro Inc.

TMSTAF vs. QA‟s Automation

6/12/2012 15| Copyright 2012 Trend Micro Inc.

QA‟s Automation Process

• Automation construction

• Automation online

6/12/2012 16| Copyright 2012 Trend Micro Inc.

Test

cases

creation

Test cases

convert to

automation

scripts

Automation

script pre-

run and

debug

Integrate

with build

release

process

Build

quality

verification

Automation

environment

setup

6/12/2012 17Confidential | Copyright 2012 Trend Micro Inc.

Test Cases Creation

• Based on concept of „test suite‟, test case can be created for both „manual‟ and „automation‟ use

6/12/2012 18| Copyright 2012 Trend Micro Inc.

6/12/2012 19Confidential | Copyright 2012 Trend Micro Inc.

Automation

Environment Setup

• Automation environment can be deployed easily

6/12/2012 20| Copyright 2012 Trend Micro Inc.

Automation Environment

Python 2.6

TMSTAF

framework

STAF 3.x

6/12/2012 21Confidential | Copyright 2012 Trend Micro Inc.

Test Cases Convert to

Automation Scripts

• Convert test case to automation scripts structurally

6/12/2012 22| Copyright 2012 Trend Micro Inc.

Automation

Module

Automation

ModuleTest step

Test stepTest case

(automation

script)

Automation

Module

6/12/2012 23| Copyright 2012 Trend Micro Inc.

6/12/2012 24| Copyright 2012 Trend Micro Inc.

6/12/2012 25Confidential | Copyright 2012 Trend Micro Inc.

3 Weapons…

I - Scenario Mechanism

6/12/2012 26| Copyright 2012 Trend Micro Inc.

var1 = 111

var2 = 222

func = PythonFunction('moduleATester', 'testFunc1')

r = self.addTest(func, [var1,var2])

r.failUnlessEqual(0)

var1 = 333

var2 = 444

func = PythonFunction('moduleATester', 'testFunc1')

r = self.addTest(func, [var1,var2])

r.failUnlessEqual(0)

test0010.py

test0020.py

6/12/2012 27| Copyright 2012 Trend Micro Inc.

var1 = 111

var2 = 222

self.setScenario('_scenario01.py')

test0010.py

test0020.py

var1 = 333

var2 = 444

self.setScenario('_scenario01.py')

func = PythonFunction('moduleATester', 'testFunc1')

result = self.addTest(func, [var1,var2])

result.failUnlessEqual(0)

_scenario01.py

II - Setup, Teardown and Test procedures

6/12/2012 28| Copyright 2012 Trend Micro Inc.

Test Case

Setup steps

Test steps

Teardown steps

6/12/2012 29| Copyright 2012 Trend Micro Inc.

• Setup = addSetup()

• Test = addTest()

• Teardown = addTeardown()

III - Result Verification

6/12/2012 30| Copyright 2012 Trend Micro Inc.

• Execution result of each steps (includes setup and teardown) can be verified easily

Passed if ‘result’ = 0

6/12/2012 31Confidential | Copyright 2012 Trend Micro Inc.

Script Pre-run and

Debug

6/12/2012 32| Copyright 2012 Trend Micro Inc.

• More efficiency development of automation

Refinement

of script

Script

execution

Debugging

of script

6/12/2012 33| Copyright 2012 Trend Micro Inc.

• Script execution from command line interface

6/12/2012 34| Copyright 2012 Trend Micro Inc.

Run Target Case

Run Target Suite

“-h” Online Help

Easy to Integrate

6/12/2012 35| Copyright 2012 Trend Micro Inc.

• Execution log for tracing/debugging

6/12/2012 36| Copyright 2012 Trend Micro Inc.

6/12/2012 37Confidential | Copyright 2012 Trend Micro Inc.

Automation Online

6/12/2012 38| Copyright 2012 Trend Micro Inc.

Build Release Flow

6/12/2012 39| Copyright 2012 Trend Micro Inc.

1

2

3

4

5

6/12/2012 40Confidential | Copyright 2012 Trend Micro Inc.

Build Quality

Verification

6/12/2012 41| Copyright 2012 Trend Micro Inc.

Mail Notifications

6/12/2012 42| Copyright 2012 Trend Micro Inc.

6/12/2012 43| Copyright 2012 Trend Micro Inc.

Detail Testing Report

6/12/2012 44| Copyright 2012 Trend Micro Inc.

6/12/2012 45| Copyright 2012 Trend Micro Inc.

6/12/2012 46| Copyright 2012 Trend Micro Inc.

6/12/2012 47| Copyright 2012 Trend Micro Inc.

6/12/2012 48| Copyright 2012 Trend Micro Inc.

6/12/2012 49| Copyright 2012 Trend Micro Inc.

6/12/2012 50| Copyright 2012 Trend Micro Inc.

6/12/2012 51| Copyright 2012 Trend Micro Inc.

6/12/2012 52Confidential | Copyright 2012 Trend Micro Inc.

So, with TMSTAF,

you can…

6/12/2012 53| Copyright 2012 Trend Micro Inc.

Test

cases creation

Test cases

convert to

automation

scripts

Automation script

pre-run and debug

Automation

environment

setup

Test case can be used for both ‘manual’ and

‘automation’ use, no extra effort needs

Easy to setup automation environment for

development and execution

Implement test cases with structural and flexible

mechanism

More efficiency development and trouble-shooting

of automation

Integrate with

build release

process

Build quality

verification

Easy integrate with standard build release process

Useful information in testing report can help to

identify issue more quickly

Q&A

6/12/2012 54| Copyright 2012 Trend Micro Inc.