pysys testframework

pysysTest Automation Introduction

• Open Source (hosted on SourceForge)

• Written in Python

• Extensible (Object Oriented)

• Cross Platform (Solaris, Linux, Windows)

• System Level

pysysTest Automation

• Dependencies• Python 2.7 (http://www.python.org)• PyWin32 Extensions (http://sourceforge.net/projects/pywin32)

• Download• (http://sourceforge.net/projects/pysys)

Dependencies and Download

http://www.python.org/

http://sourceforge.net/projects/pywin32

http://sourceforge.net/projects/pysys

pysysTest Automation Installation

• Windows• Run the PySys-win32-X.Y.Z.exe installer• Unpack the PySys-examples-X.Y.Z.zip

archive

• Unix• Unpack and install the PySys-X.Y.Z.tar.gz archive

• $ tar zxvpf PySys-X.Y.Z.tar.gz • $ cd PySys-X.Y.Z • $ python setup.py build• $ python setup.py install

• Unpack the PySys-examples-X.Y.Z.tar.gz archive

pysysTest Automation What You Get

• PySys is added as a site package to the Python install• C:\Python27\Lib\site-packages\pysys• /usr/local/lib/python2.7/site-packages/pysys

•The “pysys.py” launcher is added to the Python scripts location• C:\Python27\Scripts\pysys.py

• /usr/local/bin/pysys.py

• A PySys shortcut is added to the Windows Start Menu with links to the Uninstaller, Release Notes, and API Documentation

pysysTest Automation Project Layout

• The examples distributed with PySys highlight the typical project layout

pysys-examples │ ├── .pysysproject │ ├── fibonnaci │ ├── testcases │ │ ├── Fibonacci_test_001 │ │ │ ├── .pysystest │ │ │ ├── run.py │ │ │ ├── Input │ │ │ ├── Output │ │ │ └── Reference │ │ ├── Fibonacci_test_002 │ │ ├── Fibonacci_test_003 │ │ └── Fibonacci_test_004 │ └── utilities │ ├── internal │ ├── testcases │ └── utilities

pysysTest Automation Project Meta Information

• The .pysysproject file defines the project root location• Allows setting of site specific properties• Supports property expansion, e.g.

<pysysproject>

<property environment="env"/>

<property osfamily="osfamily"/>

<property file="${osfamily}.properties" />

<property name="versionString" value="${osfamily} (${version})"/>

<property name="user" value="${env.USERNAME}" default="Test User"/>

</pysysproject>

pysysTest Automation Project Meta Information (cont)

• The .pysysproject file allows addition of paths to the PYTHONPATH for location of site specific utilities and extensions

• Allows the specification of alternative runner, writer classes, e.g.

<pysysproject>.

.<runner classname=“MyRunner" module=“com.fredblogs.runner"/><formatters>

<!– Set the stdout output format --><formatter name="stdout" messagefmt="%(asctime)s %(levelname)-5s %(message)s" datefmt="%Y:%m:%d %H:%M:%S"/></formatters>

<!– Write output to target/pysys-reports in Junit style --><writers>

<writer classname="JUnitXMLResultsWriter" module="pysys.writer" /></writers>

</pysysproject>

pysysTest Automation Project Meta Information (conf)

• The .pysysproject file is translated to an instance of a Project class

• An instance of the Project class is added as a global constant within the framework

• By default the Project class instance has a single data attribute (root) which denotes the full path to the project, i.e. PROJECT.root

• Each property specified in the file is added dynamically as a data attribute to the class, i.e. PROJECT.user

pysysTest Automation Test Case Meta Information

• The .pysystest details the meta information of the test

• The test type (auto or manual)• The test title and description• The test state (runnable, deprecated, skipped)• The groups the test belongs to• The modes the test can run in• The test python class for execution and validation• The test input, output and reference locations• The requirements covered by the test (traceability)

pysysTest Automation Test Case Meta Information (cont)

<pysystest type="auto" state="runnable">

<!– Set the testcase description --> <description> <title>Check correct generation of fibonacci series</title> <purpose>

Uses the generation tool to generate the first ten entries of the series, writing to stdout

</purpose> </description>

<!– Set the groups the test belongs to, and modes it can run in --><classification>

<groups> <group>Fibonacci</group> </groups> <modes/> </classification>

<!– Define data input location and run class --> <data> <class name="PySysTest" module="run"/> <input path=“Input"/> </data>

<!– Define the requirements covered by the test --> <traceability> <requirements> <requirement id="AL1" /> </requirements> </traceability>

</pysystest>

pysysTest Automation The BaseTest Class

• The BaseTest class is the parent class for all test cases and provides;• Cross platform process management and monitoring• Test timing routines• Assert routines for test validation

• Test cases must extend the BaseTest class and implement the execute() and validate() methods

• Test cases may also override the setup() and cleanup() methods

• The BaseTest.cleanup() ensures all processes started during the execution of the test are terminated on completion

pysysTest Automation The BaseTest Class (cont)

from pysys.constants import *from pysys.basetest import BaseTest

class PySysTest(BaseTest):

def execute(self):self.startProcess(command=sys.executable,

arguments = ["%s/fibonacci.py" % self.input], environs = os.environ, workingDir = self.input, stdout = "%s/fibonacci.out" % self.output, stderr = "%s/fibonacci.err" % self.output,

state=FOREGROUND)

def validate(self):# first validation diffs the output with the referenceself.assertDiff('fibonacci.out‘, 'ref_fibonacci.out‘)

# next validation looks for regexps in the output fileself.assertGrep('fibonacci.out‘, expr= '34‘ )self.assertGrep('fibonacci.out‘, expr= '55‘ , contains=FALSE)

# next validation looks for an ordered sequence of regexpself.assertOrderedGrep('fibonacci.out‘, exprList=['0', '1', '1', '5‘])

# next validation looks for the number of lines matching a regexprself.assertLineCount('fibonacci.out‘, expr='^[0-9]+‘, condition='==10‘)


•Assert* methods append a validation outcome to an internal data structure of the BaseTest•Valid outcomes in order of precedence are

• SKIPPED• BLOCKED• DUMPEDCORE• TIMEDOUT• FAILED • NOTVERIFIED• PASSED

•The overall outcome of a test is determined by the highest ranking entry in the list of recorded outcomes

•Assert* methods can be called anywhere within a test case (the validate() method is more convention than enforcement)


• Elements of the .pysystest file are translated to data attributes of the BaseTest class instance on its instantiation

• self.input – the testcase input directory• self.output – the testcase output directory• self.reference – the testcase reference directory• self.mode – the mode the test is being run in• self.project – copy of the Project class

pysysTest Automation The BaseRunner Class

• The BaseRunner class is responsible for running a set of testcases, and audit trail reporting

• The BaseRunner defines methods which may be overridden by a subclass to perform operations

• setup() - before a set of tests are run • testComplete() - after a single test is run • cycleComplete() - after a cycle of tests is run • cleanup() - on completion of all tests

• A BaseRunner instance is passed a list of BaseTest instances and iterates through them calling their setup(), execute(), validate() and cleanup() methods respectively

• You can provide your own instance of the runner, e.g. to override the setup() action in order to perform pre-test actions before running a group of tests – just configure in the .pysysproject file

pysysTest Automation The PySys Launcher

• The pysys.py launcher provides the ability to;• Run a subset or group of tests• Print the meta information for a subset or group of tests• Make new tests• Clean the output of previously run tests

C:\>pysys.py --help

PySys System Test Framework (version 0.9.2)

Usage: pysys.py [mode] [option]* { [tests]* | [testId] } where [mode] can be; run - run a set of tests rooted from the current working directory make - make a new testcase directory structure in the current working directory print - print details of a set of tests rooted from the current working directory clean - clean the output subdirectories rooted from the current working directory

For more information on the options available to each mode, use the -h | --help option, e.g.

pysys.py run --help

pysysTest Automation The PySys Launcher (cont)

• The pysys.py launcher • Searches up the directory tree to find the .pysysproject file• Searches down the directory tree to find all .pysystest files• Uses the .pysystest files to create a list of BaseTest class

instances• Passes the list to an instance of the BaseRunner class to

execute the tests


The PySys Launcher (cont)

• The pysys.py run tasks allows you to• Run a single test by id

• pysys.py run Test_001

• Run a range of tests by id• pysys.py run Test_001:Test_003

• Run tests in a loop to catch non-deterministic behaviour• pysys.py run –c5 Test_001

• To include and exclude tests added to groups• pysys.py run –i smoke –e performance

• To run tests in parallel with a defined number of threads• pysys.py run –n4

• Use the –help option to get the full list – there’s lots of others!

pysysTest Automation Integration With Eclipse

• Integration with Eclipse requires the PyDev Eclipse Plugin (http://pydev.sourceforge.net)

• PySys tests are executed within Eclipse using Python Run Configuration

• The main module is set to the pysys.py launcher

• The base directory is set to be ${resource_loc} - gives The absolute path on the system's hard drive to the currently selected resource

• The program argument is set to run, print, or make

• The configuration is added to the favourites menu

http://pydev.sourceforge.net/


Integration With TeamCity

• Integration with TeamCity requires using the JUnitXMLResultsWriter in the .pysysproject file

<!– Write output to target/pysys-reports in Junit style --><writers>

<writer classname="JUnitXMLResultsWriter" module="pysys.writer" /></writers>

• In TeamCity, create a command line build step that runs the pysys.py launcher

pysysTest Automation Integration With TeamCity (cont)

• Create an XML report processing build feature to pick up the test output and display the progress at runtime

• Sit back and let TeamCity do the rest!

pysysTest Automation PySys Extensions

• PySys is designed to allow extension modules to be added to the installation

• Aim is to build up a suite of extension modules to assist in automated testing across a variety of domains

pysys testframework

Documents

test validation test

test input

test title

test traceability

test python class

test type auto

test state runnable

pysys shortcut