quality assurance software test plan · quality assurance software test plan alan greer, chris...

Project Documentation Document PROC-0015

Revision A

Quality Assurance Software Test Plan

Alan Greer, Chris Mayer & Aya Yoshimura

Observatory Sciences Ltd.

July 10, 2015

QA Software Test Plan

PROC-0015, Revision A Page 1 of 88

REVISION SUMMARY:

1. Date: 18 July 2013 Revision: Draft 1 Changes: Original Version

2. Date: 29 August 2014 Revision: Draft 2 Changes: Extensive updates following implementation

3. Date: 27 February 2015 Revision: Draft 3 Changes: Updated for release with Canary 8 including descriptions of configuration

editor

4. Date: 10 July 2015 Revision: A Changes: 3.6 Creating Testing Automation Framework Configuration File section

updated. New section 5.2.7 checkout_directories added. 5.4 Testing Automation Framework Configuration File section updated. Initial formal release.



Table of Contents

1. INTRODUCTION ................................................................................................... 5

1.1 PURPOSE ............................................................................................................... 5 1.2 SCOPE ................................................................................................................... 5 1.3 DEFINITIONS, ACRONYMS AND ABBREVIATIONS ........................................................ 5 1.3.1 Abbreviations ............................................................................................................... 5 2. SYSTEM OVERVIEW ........................................................................................... 6 2.1 INTRODUCTION ....................................................................................................... 6 2.2 IMPLEMENTATION LANGUAGE .................................................................................. 7 2.3 SOURCE CODE ....................................................................................................... 7

3. QUICK START GUIDES ....................................................................................... 8 3.1 INSTALLATION OF THE TEST FRAMEWORK................................................................ 8

3.2 WRITING AND EXECUTING A SYSTEM TEST ............................................................... 8

3.3 WRITING AND EXECUTING A JAVA UNIT TEST ........................................................... 9 3.4 WRITING AND EXECUTING A C++ UNIT TEST ........................................................... 11 3.5 INSTALLATION OF THE TAF AND E2E TEST EXECUTOR ............................................ 12

3.6 CREATING TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE ...................... 12 3.7 RUNNING TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE ........................ 13 3.8 CREATING THE ETE CONFIGURATION FILE .............................................................. 14

3.9 RUNNING THE ETE ............................................................................................... 15 4. TESTING ENVIRONMENT ................................................................................. 18

4.1 JAVA TESTS (JUNIT) ............................................................................................. 18 4.1.1 Integration of JUnit ......................................................................................................18 4.2 C++ TESTS (CXXTEST) ......................................................................................... 19 4.2.1 Integration of CxxTest .................................................................................................19 4.3 PYTHON TESTS ..................................................................................................... 19 4.3.1 TestFrameworkStep.py ...............................................................................................20 4.3.2 TestFrameworkStepQA.py ..........................................................................................20 4.3.3 TestFrameworkFailure.py ...........................................................................................24 4.3.4 TestFramework.py .......................................................................................................24 4.3.5 TestFrameworkExecutor.java .....................................................................................24 5. TESTING AUTOMATION FRAMEWORK ........................................................... 25 5.1 INSTALLATION OF THE TESTING AUTOMATION FRAMEWORK ..................................... 25

5.2 TESTING AUTOMATION FRAMEWORK PYTHON CLASSES .......................................... 26 5.2.1 TestingAutomationFramework.py ..............................................................................26 5.2.2 TestingAutomationFrameworkConfigFileReader.py .................................................27 5.2.3 TestingAutomationFrameworkBuilder.py ..................................................................27 5.2.4 TestingAutomationFrameworkExecutor.py ...............................................................28 5.2.5 taf_interactive.py .........................................................................................................28 5.2.6 TAFConfigfileReaderForInteractive.py ......................................................................29 5.2.7 checkout_directories ..................................................................................................29 5.2.8 Report Format ..............................................................................................................29 5.3 RUNNING TESTS USING TAF_INTERACTIVE ............................................................. 31 5.4 TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE ..................................... 34 5.4.1 Testing Automation Framework Configuration File Editing Tool.............................37 5.4.1.1 ‘—help’ command........................................................................................................37 5.4.1.2 ‘—list’ command ..........................................................................................................38



5.4.1.3 ‘—new’ command ........................................................................................................39 5.4.1.4 ‘—edit’ command.........................................................................................................42 5.4.1.5 ‘—check’ command .....................................................................................................43 5.4.1.6 Command to edit SITE_CONFIG items ......................................................................44 5.4.1.6.1 –siteconfcopy command ......................................................................................44 5.4.1.6.2 –siteconf command ...............................................................................................45 5.4.1.6.3 – addsiteconf command .......................................................................................45 5.4.1.6.4 – removesiteconf command .................................................................................46 5.4.1.7 –copy command ..........................................................................................................46 5.4.1.8 Interactive configuration tool configure_tests ..........................................................47 6. E2E TEST EXECUTOR ....................................................................................... 51

6.1 END TO END SIMULATOR ...................................................................................... 51 6.2 RUNNING THE ETE ............................................................................................... 51 6.3 CONFIGURATION OF THE ETE ............................................................................... 54 6.3.1 ETE Configuration File ................................................................................................54 6.3.2 Configuration Tool ......................................................................................................56 6.3.2.1 ‘--list’ command ...........................................................................................................56 6.3.2.2 ‘--new’ command .........................................................................................................57 6.3.2.3 ‘--add’ command ..........................................................................................................57 6.3.2.4 ‘--del’ command ...........................................................................................................59 6.3.2.5 ‘--addvm’ command .....................................................................................................59 6.3.2.6 ‘--delvm’ command ......................................................................................................61 6.3.2.7 ‘--list’ command ...........................................................................................................61 6.3.2.8 ‘--edit’ command ..........................................................................................................62 6.3.2.9 ‘--check’ command ......................................................................................................63 6.3.2.10 ‘--copy’ command ..................................................................................................63 6.3.2.11 ‘--help’ command ...................................................................................................63 6.3.2.12 Interactive configure tool ......................................................................................64 6.4 TEST RESULTS AND REPORT GENERATION ............................................................ 67 6.5 CONFIGURING A CRON JOB FOR ETE ...................................................................... 68

7. SETUP OF VMWARE VIRTUAL MACHINE ....................................................... 69 7.1.1 Virtual Machine Snapshot Setup ................................................................................69 7.1.2 Shared Folder ..............................................................................................................69 7.1.3 Remote Control of Virtual Machines ..........................................................................69 8. UNIT TESTING ................................................................................................... 70

8.1 WRITING A UNIT TEST ........................................................................................... 70 8.1.1 Writing a Java Unit Test ..............................................................................................70 8.1.2 Writing a C++ Unit Test ...............................................................................................71 8.2 EXAMPLE JAVA UNIT TEST .................................................................................... 72 8.3 EXAMPLE C++ UNIT TEST ..................................................................................... 74 8.4 ADDING A UNIT TEST TO A TEST SET .................................................................... 77

9. SYSTEM TESTING ............................................................................................. 78 9.1 WRITING A PYTHON SYSTEM TEST ......................................................................... 78 9.2 EXAMPLE PYTHON SYSTEM TEST ........................................................................... 79 9.3 ADDING A PYTHON SYSTEM TEST TO A TEST SET ................................................... 80

10. REGRESSION TESTING .................................................................................... 81 10.1 SETTING UP A REGRESSION TEST RUN .................................................................. 81 11. ETE TEST SERVER............................................................................................ 82

12. FUTURE UPDATES ............................................................................................ 87



13. REFERENCES .................................................................................................... 88



1. INTRODUCTION

1.1 PURPOSE

The intention of this document is to describe the structure of the software and the procedures that form the

Quality Assurance/Quality Control Software Testing for the DKIST CSF, Base, TCS and subsystems

including how this testing is executed, manually or automatically. This document also describes any

setup procedures required and how reports are generated when the tests are executed.

The intended audience of this document is:

The developers of the QA work package, and

The developers of any DKIST systems or subsystems that are required to write tests.

The layout of this document is as follows:

Section 2 provides a brief overview of the entire DKIST test system. Section 3 presents the testing

environment, describing the types of test language (Java, C++ and Python) that are used. Section 4

describes how tests are executed on a machine and the tool available to carry out this execution. Section

5 describes how multiple sets of tests can be run using a tool that coordinates the execution of test sets.

Section 6 describes the setting up of a virtual machine for use within the DKIST test system. Sections 7

and 8 describe the two types of test (Unit and System) and provide descriptions and examples of how to

write each type of test. Section 9 presents a description of a typical regression test setup, and finally

sections 10 and 11 present lists of test candidates for each type of test.

1.2 SCOPE

The software described by this design is the DKIST Quality Control Test System.

The purpose of the DKIST Test System is to provide a suite of software tests, both low and high level,

that thoroughly test all parts of the DKIST CSF and Base code. It also provides a test-framework for

integrating high level subsystem tests into the test framework. All of the individual test tools and

harnesses are integrated into a framework that can be configured to run manually or automatically (or

both) with full reporting features.

1.3 DEFINITIONS, ACRONYMS AND ABBREVIATIONS

Specific definitions, acronyms and abbreviations used in this document are described below. For a more

general glossary and acronym list as used in DKIST see [3].

1.3.1 Abbreviations

CSF Common Services Framework

E2E End-to-end simulator

ETE End-to-end simulator Test Executor

QA Quality Assurance

QC Quality Control

TAF Test Automation Framework

XML Extensible Markup Language



2. SYSTEM OVERVIEW

2.1 INTRODUCTION

The Software QA for DKIST is a framework designed to provide a set of tools for testing the DKIST

CSF, Base, TCS and subsystems. The framework provides tools for QC testing in three languages (Java,

Python and C++) at two levels (unit testing and system testing). The framework also provides all the

necessary tools to perform continuous automated regression testing and is easily configurable for

differing versions of DKIST software. The framework is designed to work on any hardware architecture

by making use of virtual machines but ultimately will execute on the DKIST End-to-end (E2E) simulator

hardware. The framework is extensible to allow future integration of subsystem tests and addition of unit

tests to verify new functionality.

Figure 1. Software Test Framework Overview.

The QA framework is split into three distinct pieces of software.

The first piece of software provides APIs to make writing tests simple and ensures all tests generate the

same style of reports. Tests can be written in Java, C++ or Python as necessary and the APIs provide

methods for verification and reporting test results. A description of the three language types and APIs can

be found in section 4.

The second piece of software is responsible for building specific versions of CSF, Base and other systems

on a machine. It is configured by supplying an XML configuration file which specifies the versions of

each system to checkout, any special build instructions required (e.g. simulated or real hardware) and also

a list of tests to run once the build has completed. This second piece of software is called the Test

Automation Framework (TAF) and is described in more detail in section 5 of this test plan. This Test

Automation Framework can be executed on a real machine or a virtual machine.

The final piece of software is responsible for deploying virtual machines and running test configurations

using the Test Automation Framework described above. This final piece of software will always be

E2E Hardware

Virtual Machine Snapshot

Test Automation Framework

Instance

E2E Test Executor



Instance



Instance



running in the background to check when new test sets should be deployed and started. It will run as a

cron job on a machine and there will be a command set to interact with it. The command set will offer

the ability to quickly add a new test set, specify how often the test set should be run and remove test sets

that are no longer required. The final piece of software is called the E2E Test Executor (ETE) and is

described in more detail in section 6.

Together these three software facilities provide a fully automated testing framework that can easily be

extended for new test use cases, additional regression test runs or for trialing new DKIST software

configurations. The use of virtual machines provides a cost saving as no new hardware is required when

an additional test configuration is added to the system. Once configured, the system can run fully

autonomously for as long as is required, with test reports logged and optionally emailed after each test run

completes.

2.2 IMPLEMENTATION LANGUAGE

To cover all types of tests and all CSF code three languages will be used to develop the required testing

framework. The languages are Java, C++ and Python. Java and C++ are required to write tests that

directly interact with corresponding language specific CSF code. Python will be adopted as a higher level

testing suite due to the CSF integrated Jython script executor. Python can also be used to carry out any

high level automation tasks required as part of the software QA.

2.3 SOURCE CODE

All source code written and developed as part of the software QA contract will be provided. The code will

be developed and committed to a separate DKIST QA repository in Tucson.

All source code shall be documented in a manner consistent with good software practices. All source

files shall contain a header containing the version number, revisions, author(s), and functional description.

All functions within a source file shall have a description of the interface and operation of the function,

and all source shall be clearly commented.



3. QUICK START GUIDES

The following sections provide step by step instructions on how to start using the QA framework, without

explaining the details of each step. For a reference guide see the sections later on in this document.

3.1 INSTALLATION OF THE TEST FRAMEWORK

Installing the testing framework into an existing source tree is straightforward. Check out from the "qas"

repository to install the framework in to the tree.

cd $ATST/..

export CVSROOT=:pserver:[email protected]:2401/home/atst/src/qas

cvs login

cvs checkout atst

cd $ATST

make build_all

The "qas" repository contains all of the framework code required to execute test scripts (Python

unit/system tests, Java unit tests and C++ unit tests) as well as an existing set of CSF and Base test scripts

written to test the CSF code itself.

3.2 WRITING AND EXECUTING A SYSTEM TEST

This section demonstrates how to write and execute an individual system test. It assumes the user already

has a fully built ASDT.

System tests written in python can be placed anywhere and the test framework simply pointed to the

script when executed. Create a new file called helloWorld.py in $ATST and enter the following into the

file.

class Step1( TestFrameworkStepQA ):

def Run( self ):

self.LogMessage("Hello World!")

TestFramework.Register("HELLO.WORLD.0001")

TestFramework.Description("The classic hello world example")

TestFramework.CvsId("$Id:$")

TestFramework.AddStep(Step1())

Save the file and then execute the test.

cd $ATST

./bin/RunSystemTest --script=./helloWorld.py --report=report.txt

As the tests logs a message this will be printed to the screen during the execution of the test.

[2014-07-17 11:04:22.638] (note:HELLO.WORLD.0001.64.1:) ATST.QA.SYSTEM:

[Step1] Hello World!

The test should complete successfully, and once it has the report generated by the test (specified by the

--report option) can be opened to see the results.

[ajg@atst01 atst]$ more report.txt

<?xml version="1.0" ?>

<TEST>

<ID>



HELLO.WORLD.0001

</ID>

<CVSID>

$Id:$

</CVSID>

<VMNAME/>

<SNAPSHOT/>

<TYPE>

system

</TYPE>

<TIME>

2014-07-17 11:04:22.636

</TIME>

<MACHINE>

atst01.observatorysciences.co.uk

</MACHINE>

<RESULT>

PASS

</RESULT>

<FAILURES/>

</TEST>

Looking again at the original test script, creating a test step is simply a case of declaring a class that is a

subclass of TestFrameworkStepQA. The created class must contain the method "def Run(self):"

but otherwise has no limitations. The "Run" method is executed by the test framework. To register the

test step class with the test framework the static method TestFramework.AddStep()is called.

There are many additional helper methods that are available to the test step class (courtesy of the

TestFrameworkStepQA class) and these methods are explained in section 4. For this simple test we made

use of the LogMessage method only.

3.3 WRITING AND EXECUTING A JAVA UNIT TEST

This section demonstrates how to write and execute an individual Java unit test. It assumes the user

already has a fully built ASDT. We will now create a simple unit test that will fail on purpose.

Unit tests written in Java must be placed under the "src/java" folder within the ASDT. Create a new file

called ExampleUnitTest.java in $ATST/src/java/atst/qas/example and enter the following into the file.

package atst.qas.example;

import static org.junit.Assert.assertEquals;

import org.junit.Test;

import org.junit.Ignore;

import org.junit.runner.RunWith;

import org.junit.runners.JUnit4;

import atst.qas.framework.ITestScript;

@RunWith(JUnit4.class)

public class ExampleUnitTest implements ITestScript

{

@Override

public String getCVSID()

{

return "$Id:$";

}



@Test

public void thisAlwaysPasses() {

}

@Test

public void thisAlwaysFails() {

assertEquals(1.0, 3.0, 0.1);

}

@Test

pubic void thisAlsoAlwyasFails() {

assertEquals(1, 3);

}

}

Java test scripts must be compiled before they can be executed.

make classes

Now the test can be executed

cd $ATST

./bin/RunJavaUnitTest --script=atst.qas.example.ExampleUnitTest --report=report.txt

Notice that a Java test is specified not by the path to the file but instead by the fully qualified class name

after compilation. As the Java unit tests are built using JUnit all of the available features of that test

framework are available when writing the QA unit tests. The main addition to the QA unit tests is the use

of a custom set of runner classes that produce the correct format for the test report file, and the use of the

ITestScript interface which ensures the CVS ID of the test script can be included in the report, which

provides a very useful method for getting back to the correct version of the test script from a test report

file. A full description of the classes used for the Java unit tests can be found in section 4.

The report file generated in this case is shown below.

[ajg@atst01 atst]$ more report.txt

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<TEST>

<ID>ExampleUnitTest</ID>

<CVSID>$Id: ExampleUnitTest.java,v 1.1 2014/02/26 13:34:38 AlanGreer Exp $</CVSID>

<VMNAME/>

<SNAPSHOT/>

<TYPE>unit</TYPE>

<TIME>2014-08-26 08:24:27.897</TIME>

<MACHINE>altair</MACHINE>

<RESULT>FAIL</RESULT>

<FAIL_LOCATION>thisAlwaysFails(atst.qas.example.ExampleUnitTest)</FAIL_LOCATION>

<FAIL_REASON>expected:[1.0] but was:[3.0]</FAIL_REASON>

<FAIL_LOCATION>thisAlsoAlwaysFails(atst.qas.example.ExampleUnitTest)</FAIL_LOCATION>

<FAIL_REASON>expected:[1] but was:[3]</FAIL_REASON>

</TEST>

The output is in the same format as that of the python tests. Note that there is a single <RESULT> entry

showing whether the overall test passed or failed along with separate locations and reasons for the

individual tests. If no entry is present for a specific test then that test has passed.



3.4 WRITING AND EXECUTING A C++ UNIT TEST

This section demonstrates how to write and execute an individual C++ unit test. It assumes the user

already has a fully built ASDT. We will now create a simple unit test that will fail on purpose.

Unit tests written in C++ must be placed under the "src/c++" folder within the ASDT. C++ unit tests

must be built in a special way and so it is necessary to add an additional line to the Makefile.gcc file in

the directory where any C++ unit tests are to be compiled. Edit the Makefile.gcc file in the

$ATST/src/c++/atst/qas/tests/example directory and you will notice the line

include ${ATST}/src/c++/atst/qas/framework/Makefile.gccQAS

This line includes the necessary build rules for compiling C++ unit tests and must be added to any

Makefile.gcc that wants to build tests. Now create a new file called HelloWorld.h in

$ATST/src/c++/atst/qas/tests/example and enter the following into the file. Note the file is a header (.h)

file and not a C++ source file.

#ifndef HELLOWORLD_H_

#define HELLOWORLD_H_

#include "cxxtest/TestSuite.h"

#include <fstream>

#include <iostream>

std::string CVS_ID = "$Id:$";

class HelloWorld : public CxxTest::TestSuite {

public:

void testAlwaysFails(void)

{

// Verify received has been set to true

TS_ASSERT_EQUALS(false, true);

}

};

#endif //HELLOWORLD_H_

It is then necessary to edit the Makefile.gcc file and notify the build system that this test is to be compiled.

Add the following line into the Makefile.gcc:

UNIT_TESTS += HelloWorld

Now make the directory and the test will be compiled. In C++ test files are compiled into executable test

applications that can be executed using the RunC++UnitTest script available from the $ATST/bin

directory.

$ATST/bin/RunC++UnitTest –script=atst/qas/tests/example/HelloWorld --

report=report.txt

A full description of how the C++ unit tests are integrated into the QA system can be found in section 4.

The test output for this test is shown below



<?xml version="1.0" encoding="UTF-8"?>

<TEST>

<ID>HelloWorld.h</ID>

<CVSID>$Id:$</CVSID>

<VMNAME/>

<SNAPSHOT/>

<TYPE>unit</TYPE>

<TIME>2014/08/26:08:38:59.517 GMT</TIME>

<MACHINE>altair</MACHINE>

<RESULT>FAIL</RESULT>

<LOCATION>HelloWorld.h[20]</LOCATION>

<REASON>Error: Expected (false == true), found (false != true)</REASON>

</TEST>

As for the JUnit tests an overall pass/fail is provided by the <RESULT> field along with the location and

reason for each failure.

3.5 INSTALLATION OF THE TAF AND E2E TEST EXECUTOR

The Testing Automation Framework and E2E Test Executor are stored in a separate module in the CVS

repository in Tucson and can be checked out independently from any DKIST software development tree.


cvs login

cvs checkout atst_test

Set the ATST_TEST environment variable to the directory where the atst_test has been checked out.

Build the software as follows.

cd $ATST_TEST

./admin/createDevel --make-all

make install_scripts

After the build the TAF, ETE and configuration file tools will be installed in $ATST_TEST/bin.

The TAF makes use of the python program pexpect. This can be installed with yum.

yum install pexpect

VMWare Workstation is required to run the ETE.

3.6 CREATING TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE

A TAF configuration file is required to run the TAF. The configuration editing tools can be used to create

or edit the file. There is a file ($ATST_TEST/src/python/editor/taf_defaults) which stores default values

for the TAF configuration file to be used by the tools. Make sure to change ‘INST_DIR’ value to the

directory where the ATST Software Development Tree is installed. (This is the directory that contains the

ASDT. The TAF sets the ATST environment variable to <INST_DIR>/atst. See 5.4 for more details.) Set

the ‘REPORT’ value to a directory where the reports from the TAF should be kept.

INST_DIR=<Path to the parent directory of ‘atst’>

REPORT=<Existing directory where the TAF reports should be stored>

After editing the taf_defaults file, run configure_taf to create a new TAF configuration file.



cd $ATST_TEST/bin

./configure_taf --new new_taf.xml --id="New TAF configure file"

New TAF configure file (new_taf.xml) created.

In this example we’ll use the atst which is already installed and will run a system test that has been

installed with the Test Framework. Add a test

$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py

with index of ‘1’ to the configure file using configure_taf. Also make sure that INITDB and

INSTALL_APP are set to ‘No’.

./configure_taf --edit new_taf.xml --

add_python_system_test='$ATST'/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101

.py --add_python_system_id=1 --initdb=No --install_app=No

TAF config file (new_taf.xml) is saved.

./configure_taf --list new_taf.xml

<<<< new_taf.xml START >>>>

ID:New TAF configure file

Install directory:/opt

Report file directory:/mnt/hgfs/Reports

System tests:

PYTHON id=1 $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py

Install application:no

Postgresql name:postgresql-9.3

init-db:no

Start ICE service:Yes

<<<< new_taf.xml END >>>>

3.7 RUNNING TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE

The TAF is executed as follows.

./TAF --tafconfig=new_taf.xml

After the TAF completes, there should be a report file in the directory specified in the TAF configuration

file.

cd /mnt/hgfs/Reports/2015_02_25_12_16_21/

more SYS_REQ_4.1_0101_PY_20150225121626.xml


<TEST>

<ID>

SYS_REQ_4.1_0101

</ID>

<CVSID>

$Id: SYS_REQ_4.1_0101.py,v 1.5 2014/09/10 09:20:43 ChrisMayer Exp $

</CVSID>

<DESCR>

Testing deployment of containers.

</DESCR>

<VMNAME/>

<TAF_ID>

New TAF configure file

</TAF_ID>

<SNAPSHOT/>



<TYPE>

system

</TYPE>

<TIME>

2015-02-25 12:16:31.748

</TIME>

<MACHINE>

qas01

</MACHINE>

<RESULT>

PASS

</RESULT>

<FAILURES/>

</TEST>

If running only one test at a time, then TAF_interactive should be used. This program also requires a TAF

configuration file and executed as follows.

./TAF_interactive new_taf.xml

$ATST is not set. INST_DIR in the TAF configure file is set to /opt.

Use /opt/atst as $ATST? (Y/n) >y

/opt/atst is set to $ATST.

======================= menu ======================

1:Run system tests from a list.

2:Run Java unit tests from a list.

3:Run CPP unit tests from a list.

4:List system tests.

5:List Java unit tests.

6:List CPP unit tests.

7:Run system test (by test file path).

8:Run Java unit test (by class name).

9:Run CPP unit test (by class path).

99:Exit this program.

===================================================

With this program a test to execute can be specified by its index, and it can also execute tests that are not

listed in the configure file.

3.8 CREATING THE ETE CONFIGURATION FILE

An ETE configuration file is required to run the ETE. The configuration editing tools can be used to

create or edit the file. There is a file ($ATST_TEST/src/python/editor/ete_defaults) which stores default

values for the ETE configuration file to be used by the tools. Setting appropriate values in this file will

make editing an ETE configuration file easier.

In this example, we assume that we have a virtual machine

(/drive/VMWareMachines/ATST_READY/ATST_READY.vmx) with a snapshot (Installed) which was

taken after the atst was installed and made ready to run system tests. We will run the TAF on the virtual

machine with a TAF configuration file (/drive/ATST_TEST/atst_test/bin/new_taf.xml). We also assume

that the virtual machine has a directory (/mnt/hgfs/Reports) for the TAF report files and that is shared to

the host machine’s directory (/drive/ATST_TEST/Reports). Also there is a directory

(/drive/ATST_TEST/FinalReport) to store the final report of the ETE. For this example, the ete_defaults

file can be updated as shown below.



more ../src/python/editor/ete_defaults

VM_FILE=/drive/VMWareMachines/ATST_READY/ATST_READY.vmx

TAF_CONFIG=/drive/ATST_TEST/atst_test/bin/new_taf.xml

REPORT_DIR_PATH_ON_VM=/mnt/hgfs/Reports

REPORT_DIR_PATH_ON_HOST=/drive/ATST_TEST/Reports

FINAL_REPORT_DIR=/drive/ATST_TEST/FinalReport

#[email protected]

SNAPSHOT=Installed

#SYSTEM_TESTS=all

#JAVA_UNIT_TESTS=all

#CPP_UNIT_TESTS=all

A new ETE configuration file is created as follows.

./configure_ete --new new_ete.xml

Creating a new empty ETE config file...

New ETE config file (new_ete.xml) created.

Please note that the file at this moment is empty. Now add an entry for the test to run at 09:00 UTC.

./configure_ete --add new_ete.xml --time=09:00

Adding a new test instance...

VM order 1 with default values is added.

Default final report directory is used.

ETE config file (new_ete.xml) is saved.

-----ETE configure file - new_ete.xml -----

Time : 09:00

Final report directory: /drive/ATST_TEST/FinalReport

Run order: 1

VM file: /drive/VMWareMachines/ATST_READY/ATST_READY.vmx

Snapshot name: Installed

TAF config file: /drive/ATST_TEST/atst_test/bin/new_taf.xml

Report directory path on the virtual machine: /mnt/hgfs/Reports

Report directory on the host machine: /drive/ATST_TEST/Reports

---------------end-------------------

Note that it associated a test instance with a VM instance (we can have multiple instances in a test) using

the values from the ete_defaults file.

3.9 RUNNING THE ETE

The ETE requires –eteconfig parameter to specify the ETE configure file, --vmuser to specify the user of

the virtual machine and –vmpassword to specify the password for the user. More options might be

required depending on what is set in the TAF configuration file (See 6.2). The ETE is intended to be

called by a cron job (see 6.5). A test instance is executed only if the current time is equal to the time that

is specified in the test instance. For example, the test instance created in the previous example wouldn’t

run unless the ETE is called at 09:00 UTC. To run the test instance at any time, we can specify –testtime

option with a desired time. Before executing the ETE, make sure the virtual machine is not running – the

ETE won’t run the test if it is.

./ETE --eteconfig=new_ete.xml --vmuser=atst-qas --vmpassword=<password>



--testtime=09:00

After the ETE completes, there should be a final report file created in the directory specified in the ETE

configuration file as Final report directory.

cd /drive/ATST_TEST/FinalReport/2015_02_25_13_45_41/

more new_ete_final_report.txt

Test Execution: Wed Feb 25 13:45:41 2015

Virtual machine: /drive/VMWareMachines/ATST_READY/ATST_READY.vmx

TAF config file: /drive/ATST_TEST/atst_test/bin/new_taf.xml

Results Summary:

Run Pass Fail

Unit 0 0 0

System 1 1 0

Details of failures:

There will also be a TAF report file for the system test in the Report directory.

cd /drive/ATST_TEST/Reports/2015_02_25_13_48_37

more SYS_REQ_4.1_0101_PY_20150225134842.xml


<TEST>

<ID>

SYS_REQ_4.1_0101

</ID>

<CVSID>


</CVSID>

<DESCR>

Testing deployment of containers.

</DESCR>

<VMNAME>

/drive/VMWareMachines/ATST_READY/ATST_READY.vmx

</VMNAME>

<TAF_ID>

New TAF configure file

</TAF_ID>

<SNAPSHOT>

Installed

</SNAPSHOT>

<TYPE>

system

</TYPE>

<TIME>

2015-02-25 13:48:49.345

</TIME>

<MACHINE>

qas01

</MACHINE>

<RESULT>

PASS

</RESULT>

<FAILURES/>



</TEST>

[atst-qas@osllx25 2015_02_25_13_48_37]$



4. TESTING ENVIRONMENT

4.1 JAVA TESTS (JUNIT)

For the Java unit tests the open source test framework JUnit has been selected for use. JUnit is actively

maintained (version 4.11 released November 2012) and has many tutorials as well as an online Javadoc

for its API. The framework can be used to automate the execution of unit tests and is easily extensible.

Tests are written for JUnit by annotating methods with the @org.junit.test annotation. This identifies to

JUnit that this method is a test method. Other annotations are available to simplify the test classes,

including the ability to execute methods before and/or after each test method, and to execute methods

once before and/or after all test methods. Test methods can be ignored if necessary and timeouts for tests

can be set. JUnit also provides other classes that help writing tests, including static assertion methods,

variable checking and comparisons.

4.1.1 Integration of JUnit

It is necessary for the DKIST testing framework to extend the capability of JUnit so that reports can be

generated that adhere to the required report format (see section 5.2.5). To integrate JUnit tests into the

DKIST test framework an additional class (JUnitATSTTestRunner.java) is required that can be executed

from the command line.

$ATST/bin/ajava atst.qas.framework.JUnitATSTTestRunner

This command is executed by the RunJavaUnitTest script introduced in Section 3.3.

The JUnitATSTTestRunner class accepts the following command line parameters:

Parameter Required/Optional Description

script Required Fully qualified test class file to execute.

report Optional Location of the report file that is to be generated for this

test script.

config Optional Not in use.

vmname Optional Name of the virtual machine that this test is running on.

snapshot Optional Name of the snapshot of the virtual machine that this test

is running on.

waitforuser Optional Not in use.

Once executed the class creates the test report file and populates it with the ID, type of test, time of test

execution, machine name that the test is executing on and the versions of CSF and Base software. It then

uses the script command line parameter to obtain the test class file and pass this into the core JUnit test

class, which executes the test and returns the results object. This class parses the results object to obtain

the rest of the information required for the test report. It is the responsibility of the TAF (see section 5) to

provide the required command line parameters for this class.

As well as the JUnitATSTTestRunner class there is also a TestReportGenerator class that is used to

generate the XML report file and an ITestScript interface. All test scripts must implement this interface

so that the test runner class can obtain the CVS version number of the file.



4.2 C++ TESTS (CXXTEST)

The C++ tests will be written to follow as closely as possible the corresponding Java tests for the unit

tests. It is essential to prove the same adherence to requirements in both languages. In cases where it is

not possible to perform the same tests due to the underlying software architecture (e.g. C++ thread model)

then the tests specific to a single language will be clearly marked as such.

For the C++ unit tests the open source test framework CxxTest will be adopted. CxxTest is actively

maintained (version 4.3 released July 2013), has extensive documentation and has already been used for

developing DKIST CSF C++ unit tests in early versions. It is easy to use because it does not require pre-

compiling a library and supports a flexible form of test discovery. It is described as being similar to

JUnit. Using CxxTest involves writing test cases that are defined in C++ header files, which are then

parsed to automatically generate test runners. It is possible to specify the class of runner or a template file

when generating the test files that allow custom code to be executed.

CxxTest provides assertions, fixtures and mocks for test cases, and the resulting compiled CxxTest files

are executables that can be run from the command line.

4.2.1 Integration of CxxTest

It is necessary for the DKIST testing framework to extend the capability of CxxTest so that reports can be

generated that adhere to the required report format (see section 5.2.5). To integrate CxxTest tests into the

DKIST test framework a template file will be used (CxxUnitATSTRunner.h). The CxxUnitATSTRunner

template will define a main entry point for the tests which accepts some additional command line

parameters:


script Optional Not in use.

report Optional Location of the report file that is to be generated for this

test script.

vmname Optional Name of the virtual machine that this test is running on.

snapshot Optional Name of the snapshot of the virtual machine that this

test is running on.

Once executed the template main creates the test report file and populates it with the ID, type of test, time

of test execution, machine name that the test is executing on and the versions of CSF and Base software.

It then executes the test listener which will be a custom class to ensure the rest of the test data is appended

to the report in the correct format. It is the responsibility of the TAF (see section 5) to provide the

required command line parameters for this class.

4.3 PYTHON TESTS

The test-framework is a set of Python classes that provide an easy to use API for writing system tests.

The classes will be adapted from the current set of test-framework classes used for the TCS acceptance

tests. The classes are executed using the available CSF Jython interpreter and so have full access to the

Java CSF services and public API. This provides the ideal method for system tests that can interact with

CSF applications through the same set of public interfaces as other CSF applications (including the JES).

The test classes simply wrap the available CSF API with additional handlers to automatically catch

exceptions and ensure test data is logged as expected. The test-framework is executed from within a CSF

Java VM and so one additional Java class is required to start the required CSF standalone application.

This is explained in more detail below.



The test-framework provides the ability to log messages throughout the test execution and these messages

are all logged to files. The test-framework also logs a report in the format required by the TAF (see

section 5.2.5).

The following sections list the classes that form the test-framework and provide a description of each

class.

4.3.1 TestFrameworkStep.py

This is a Python class. This forms the base class for developers of test-framework scripts. It contains a

small set of methods, the minimum required set for writing a test script. The following API is provided

by this class:

Method Description

AddFailure Parameter:reason - The reason for the failure

Parameter:step - The test step

Parameter: loc - The location of the failure.

This method is internal to the test framework and should not be called or

overridden. When a test fails the location and reason for the failure is added as

information by calling this method.

LogMessage Parameter: message - The message to log.

Logs a message using the CSF Log service and also to the test data file. The

message is prefixed with the unique test ID and step for future identification.

Execute Parameter: test

Parameter: step

Parameter: data

Parameter: failures

Parameter: logdb

Parameter: test_data

This method is internal to the test framework and should not be called or

overridden. The execute method is called by the test framework for each test

step and manages the calls to the "Run" and "Cleanup" methods (see below). It

also is responsible for catching exceptions that are generated by the test scripts.

Run This method is empty and it is expected to be overridden by the individual test

step classes. It is called by the "Execute" method when the test is executed. If

this is not overridden then it raises a NotImplementedError.

Cleanup This method is empty and it can be overridden by the individual test step

classes. It is called by the "Execute" method after the test step has finished.

This method will always be called, whether the test has passed or not and is

intended to give test authors the potential to clean up resources, even if a test has

failed or some other unexpected exception has occurred.

4.3.2 TestFrameworkStepQA.py

This is a Python class sub classed from TestFrameworkStep. Developers of system test-framework

scripts will subclass this and make use of the helper methods provided. This class provides hooks into all

of the CSF services. It also provides helper methods for carrying out certain tasks as well as the ability to

check for standard CSF responses and error messages. By using the helper methods the correct format of



reports are generated for the tests which can be parsed by the TAF. The following public API is provided

by this class:

Method Description

Fail Parameter: message - A message to log as the reason for the failure.

Parameter: loc - An optional parameter that specifies where the test has failed.

This forces the test step to fail and stop any further execution. It will result in an

instance of the TestFrameworkFailure class being thrown. A message is passed

to this class to explain why the test has failed.

AllFail Parameter: message - A message to log as the reason for the failure.

Parameter: loc - An optional parameter that specifies where the test has failed.

This forces the test step to fail and stop any further execution. Unlike Fail

however it also cancels any subsequent steps. It results in an instance of the

TestFrameworkFailure class being thrown. A message is passed to this class to

explain why the test has failed.

Store Parameter: attribute - An DKIST attribute to store that is available to all further

test steps that are to be executed.

The test-framework has access to all CSF services, including the CSF cache.

This method allows test steps to store values into the cache for future use. These

values are preserved across test steps.

Retrieve Parameter: name - The name of an DKIST attribute that was stored previously.

The test-framework has access to all CSF services, including the CSF cache.

This method allows test steps to retrieve previously stored values from the

cache.

StoreObject Parameter: name - A name that can be used to reference the object

Parameter: o - an arbitrary object that can be accessed by later test steps

This allows any test data that is required by a test to be stored and then retrieved

by name by later test steps

RetrieveObject Parameter: name - The name of an object that was stored previously

Look up and return test data that has been stored by a previous test step

AsyncSubmit Parameter: controller - The name of the controller to submit to.

Parameter: configuration - An DKIST configuration to submit to the controller.

Submit a configuration to a controller and return immediately with the response

code.

AsyncDelayedSubmit Parameter: controller - The name of the controller to submit to.


Parameter;delay - Delay in seconds

Submit a configuration to a controller after a specified delay and then return

immediately with the response code.

SyncSubmit Parameter: controller - The name of the controller to submit to.


Submit a configuration to a controller and wait until the controller reports that it



is complete before returning with the response code and any error code.

ParseResponse Parameter: response - The response code to parse.

Parse a response code into a string message.

GetErrorMessage Parameter: configurationID - The ID of the submitted configuration.

If a submit failed then query the error message string returned from the

controller.

N.B. Not yet fully implemented

SetValues Parameter: name - The name of the component/controller to set.

Parameter: table - An DKIST attribute table to send to the component/controller

with attributes containing values to set.

Connect to a component/controller and issue a set.

GetValues Parameter: name - The name of the component/controller to get from.

Parameter: table - An DKIST attribute table containing empty attributes with the

names of values that should be retrieved.

Connect to a component/controller and issue a get, returning an attribute table.

GetValue Parameter: cname - The name of the component/controller to get from.

Parameter: aname - The name of the attribute for the value that should be

retrieved.

Connect to a component/controller and issue a get for a single attribute returning

the value as a string

GetHealth Parameter: name - The name of the component/controller to retrieve the health

from.

Retrieve the current health of a component/controller as a string i.e. GOOD,

ILL, BAD etc.

GetLifecycle Parameter: name - The name of the component/controller to retrieve the

lifecycle from.

Retrieve the current lifecycle state of a component/controller as a string

Post Parameter: name - The name of the event channel to post on.

Parameter: table - An DKIST attribute table containing the data to post.

Post an event immediately.

AsyncPost Parameter: name - The name of the event channel to post on.

Parameter: table - An DKIST attribute table containing the data to post.

Parameter: delay - The time in seconds to delay before posting the event.

Post an event in the future. Processing continues which allows the test time to

listen for the same event.

Listen Parameter: event - The name of the event channel to listen to.

Parameter: items - A list of specific attributes to listen for.

Parameter: wait - A maximum time in seconds to wait for the event. Default is

5.0 seconds.

Listen for a specific event and return an attribute table containing the items



requested

ListenWithFilter Parameter: event - The name of the event channel to listen to.

Parameter: items - A list of specific attributes to listen for.

Parameter:filter - A list of rules. For example, [["__.name","CMP2"],

["category","alarm"]]

Parameter: wait - A maximum time in seconds to wait for the event. Default is

5.0 seconds.

Listen for a specific event with specific contents and return an attribute table

Alarm Parameter: atype - The type of alarm to raise.

Parameter: reason - The reason for raising the alarm.

Raise an alarm.

ListenForAlarm Parameter: cname - The name of the controller that the alarm originates from.

Parameter: atype - The type of the alarm to listen for.

Parameter: wait - The maximum time in seconds to wait for the alarm. Default

is 5.0 seconds.

Listen for a specific alarm to be raised.

Uninit Parameter: name - The name of the component/controller to uninit.

Helper method for un-initializing a component or controller.

VerifyLoaded Parameter: name - The name of the component/controller to verify.

Parameter: timeout - The timeout in seconds. This parameter is optional. If none

is supplied then the default is 180

Return: status - Set to 0 if successful. Non-zero otherwise

Check the lifecycle state of the specified component/controller and if necessary

change the lifecycle to "loaded".

VerifyInitialized Parameter: name - The name of the component/controller to verify.





change the lifecycle to "initialized".

VerifyRunning Parameter: name - The name of the component/controller to verify.





change the lifecycle to "running".

SearchLog Parameter: whereclauses - A list of search parameters used to query the

database. It is a list of tuples of three items. For example:

[["category","=","HEALTH"], ["time_stamp",">","2014-02-24 16:00:00.000"]]

All of these will be concatenated with AND



This is a helper method for checking log messages that have been stored in the

log database. It allows any search string to be specified and returns a list of

objects that contain all of the log entries that satisfy the search criteria.

LoadAppData Parameter: csvfile - name of file to load

Uses AppReader to load an application database file

4.3.3 TestFrameworkFailure.py

This is a Python class. This is an internal class used by the test-framework to check for failures that are

caused by unexpected problems with the test scripts. Developers of test scripts will check for pass or fail

criteria but sometimes the test script itself might fail, due to generated exceptions or unexpected data.

This class is used when catching those types of problems.

4.3.4 TestFramework.py

This is a Python class. This class is the main python class that derivatives of TestFrameworkStep.py can

register with. This class stores data relating to the individual steps, such as the time of execution and

unique ID, and executes the tests recording the output to the specified report files. This class will handle

any types of failure (expected failure or unexpected test failure) and ensure that the test-framework is able

to continue.

4.3.5 TestFrameworkExecutor.java

This is a Java class. This class will provide the command line interface for running tests using the test-

framework. The class accepts parameters which point it to a test script file and the location of test reports

and test data files. It starts up the Java CSF services using the Standalone API with a full set of service

tools, and then creates an instance of the Jython executor. Once the Jython executor is ready the python

script is loaded and executed. The test report location is passed to the TestFramework python class which

handles writing the report and test data files.



5. TESTING AUTOMATION FRAMEWORK

The Testing Automation Framework (TAF) is responsible for coordinating a large set of tests. It consists

of a set of Python scripts, Java unit tests and CPP unit tests that are executed on the (virtual) machine

where the tests are to be run. The TAF will run under the assumption that tests are to be executed from a

clean checkout. It will check its configuration file to find out which version or versions of the CSF, Base

and subsystems should be checked out. The TAF will then check out the versions into the designated

directory and perform all installation steps required to have a fully operational CSF installation.

Once the installation is complete the TAF will run in sequence all of the pre-selected system tests,

followed by the unit test. The tests that execute on a particular instance of the TAF are entirely

configurable, and as multiple TAF instances can run simultaneously within separate virtual machines it is

conceivable that several versions of CSF can be regression tested at the same time (subject to the physical

memory constraints of the underlying hardware).

It is possible to configure an instance of the TAF to not execute any tests at all, simply checkout the

required version of CSF, Base and the subsystems. This would be useful for a situation where tests

executing under another TAF instance on a separate virtual machine expect to connect to remote

application instances. A set of virtual machines can be configured to mimic a fully networked system

ready for network tests. This of course would not be suitable for testing the network architecture, only for

testing the software behavior within a distributed deployment.

The TAF is responsible for collecting all of the reports from the tests and producing the final report which

will be saved to a specified location and emailed to a specified list of addresses. The TAF is not

responsible for sending the emails containing the reports; that is the responsibility of the ETE.

5.1 INSTALLATION OF THE TESTING AUTOMATION FRAMEWORK

The testing automation framework (TAF) is stored a separate module in the CVS repository in Tucson.

As such it can be checked out independently from any DKIST software development tree (ASDT). The

instructions below assume that the software is to be checked out into a directory $ATST_TEST.

cd $ATST_TEST/..


cvs login


Once checked out the software can be built as follows

cd $ATST_TEST



After the build the TAF script will be installed in $ATST_TEST/bin.

Note that the TAF makes use of the python program pexpect. This can be installed with yum.

yum install pexpect

The TAF is executed as follows:

export ATST_TEST=<path to ATST TEST installation>

$ATST_TEST/bin/TAF --tafconfig=<TAF configuration file>



The TAF configuration file is an xml file. Its format is described in Section 5.3. The TAF script and

underlying classes can be passed a number of options but only the tafconfig option is compulsory. Other

options are described in Section 5.2.1 below.

The tests in the TAF configuration file can be executed interactively using TAF_interactive. See section

5.3 for more details.

[atst-qas@osllx25 bin]$ ./TAF_interactive TAF.xml


Use /opt/atst as $ATST? (Y/n) >


======================= menu ======================











===================================================

5.2 TESTING AUTOMATION FRAMEWORK PYTHON CLASSES

5.2.1 TestingAutomationFramework.py

This class provides the entry point for the TAF and parses the command line parameters. It coordinates

and monitors the other classes to ensure the checkout, build and test execution is executed, and it ensures

that if something goes wrong the test sequence doesn't halt prematurely. The only compulsory command

line parameter is the location of the TAF configuration file. This file is then parsed by the class

TestingAutomationFrameworkConfigFileReader described below.

The options accepted by this class are as follows:


tafconfig Required The TAF config file and path

cvsuser Optional Set the CVS user if it is required to check out an ATST

module rather than use a currently checked out location

cvspass Optional Set the CVS password associated with the given CVS

user

vm Optional Set the virtual machine name to be recorded in the test

report

snapshot Optional Set the snapshot name to be recorded in the test report

waitforuser Optional Specify this option to run tests step by step. TAF will wait

for user’s input before running a test and before each test

step.

systemtests Optional Specify Ids of the system tests to run, separated by ‘,’.

‘all’ can be set to run all the tests.

Steps of each test to be executed can also be specified

inside []. For example to execute a test (id=1), steps from

1 to 10, this option should be set to –systemtests=1[1-10].



N.B. If any of the options systemtests, javaunittests and

cppunittests is specified, then TAF will execute only the

tests that are specified. If none of these three options is

specified, then TAF will execute all the tests listed in the

configure file.

javaunittests Optional Specify Ids of the Java unit tests to run, separated by ‘,’.


cppunittests Optional Specify Ids of the Java unit tests to run, separated by ‘,’.


help Optional Display help on configuration options. The script will

then exit

5.2.2 TestingAutomationFrameworkConfigFileReader.py

The TestingAutomationFrameworkConfigFileReader class is responsible for parsing the TAF

configuration file and executing the corresponding build and execution procedures according to that file.

The class will search through all of the versions of software packages defined and invoke the builder class

below to first checkout all of the software, and then build and install it. Once this has completed it will

parse the configuration file for test script definitions and execute those using the executor class described

in Section 5.2.4.

The full list of tags that can be present in the configuration file can be found in Section 5.3.

5.2.3 TestingAutomationFrameworkBuilder.py

This class is responsible for checking out, configuring and building the specified versions of DKIST

software. The class provides helper methods for these checkout and compilation tasks. The table below

describes the public methods.

Method Description

checkout Parameter repofolder: The directory containing the CVS repository

Parameter modulname: The name of the module to be checked out

Parameter version: The version of the module to be checked out

Checks out a software module using cvs.

configsite Parameter site_config_temp: The site config template file

Parameter site_config: The site configuration file

Parameter configlist: List of config file variables and values

Sets specific variables within the site.config file.

createDevel Parameter param: Option to createDevel script

Executes the createDevel administration script with teh supplied option.

Currently accepted values are --init-bin, --init-data, --init-doc, --init-src, --

init-tools, --make-all

initdb Helper method, calls createDevel with --init-db.

checkPostgresqlService Parameter posgresql_name: Name of postgreqql service

Checks the status of the postgresql service using the name supplied in order to

check if it is runnin

startIceServices Checks if ICE services are running



make Parameter param: Specifies what is to be made

Performs a make command. The method accepts a variable which is passed to

the make command, specifying what should be built.

5.2.4 TestingAutomationFrameworkExecutor.py

This class is responsible for executing tests. It provides three methods to make test execution simple to

setup.

Method Description

runJavaUnitTest Parameter install_loc: Location of the DKIST development tree

Parameter test; Name of Java test class

Parameter result: Name of output test report file

Parameter config: Name of configuration

Parameter vmname: Name of virtual machine

Parameter snapshotname: Name of snapshot

Parameter waitforuser: ( not used)

Parameter tafid: ID of the configuration

The method executes a Java unit test

runCPPUnitTest Parameter install_loc: Location of the DKIST development tree

Parameter test; Name of C++ test executable





Parameter waitforuser: Whether to wait for user input (not used)


The method executes a C++ unit test

runSystemTest Parameter install_loc: Location of the DKIST development tree

Parameter test; Name of jython test file





Parameter waitforuser: Whether to wait for user input


Parameter steps: Steps to run

The method executes a system test

5.2.5 taf_interactive.py

This is a program to execute tests listed in TAF configure file interactively. To run this program, specify a

TAF configuration file. When this program starts up the configuration file is parsed by

TAFConfigfileReaderForInteractive.py and obtains a list of the tests specified in the file. This program

has a list of command options to run tests interactively. See Section 5.3.



5.2.6 TAFConfigfileReaderForInteractive.py

The TAFConfigfileReaderForInteractive class is responsible for parsing the TAF configuration file and

obtaining test information for the taf_interactive.py.

5.2.7 checkout_directories

The checkout_directories entry specifies the list of directories to checkout for each module. The TAF

reads this file to decide which directories to checkout. Currently the file is set as follows:

DEFAULT{

atst/resources/screens/$MODULE

atst/resources/images/$MODULE

atst/resources/$MODULE

atst/src/java/atst/$MODULE

atst/src/c++/atst/$MODULE

atst/src/idl/atst/$MODULE

atst/src/jython/atst/$MODULE

atst/doc/guides/$MODULE

atst/doc/uml/$MODULE

atst/admin/$MODULE

atst/tools/$MODULE

}

osl

{

atst/src/c++/$MODULE

}

The format is <module name>{list of directories (one in each line)}. The ‘DEFAULT’ entry is a special

entry – it is used as a default set of directories to checkout when the TAF checks out a module. The

$MODULE text will be replaced with the module name. When there is an entry of a specific module – in

this case ‘osl’ – the TAF checks out only the directories specified in the entry for the module. In this

example, for the ‘osl’ module, it will only checkout ‘atst/src/c++/osl’. With this file it is possible to

change the list of directories to checkout without changing the underlying python source code.

5.2.8 Report Format

Every test that is executed under the TAF must produce a report in the correct format so that it can be

parsed and added to the global report. The format required is xml. Below is a description of the elements

that can form the report.

Element Required Multiple Description

<test> Yes No Parent element, contains all other test report elements.

<id> Yes No Unique identifier for the test.

<type> Yes No Unit or system test.

<time> Yes No Timestamp for when the test was executed.

<machine> Yes No Name of the machine that the test was executed on.

<version> No Yes Version information for the test. This can include

versions of any of the CSF software or subsystems.

<result> Yes No This should be set either to SUCCESS or FAIL.

<fail_location> No Yes This contains information relating to where the test

failed. Multiple entries are possible to allow a



complete description of the location if available.

<fail_reason> No Yes The reason for failure of the test.

An example report is shown below. In this example the test has failed.


<TEST>

<ID>

SYS_REQ_4.1_0550

</ID>

<CVSID>

$Id: SYS_REQ_4.1_0550_2.py,v 1.1 2014/03/24 15:44:29 AlanGreer Exp $

</CVSID>

<DESCR>

Verify the controller action queue satisfies requirement 4.1-0550.

</DESCR>

<VMNAME>

/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx

</VMNAME>

<TAF_ID>

HEAD version install and Test

</TAF_ID>

<SNAPSHOT>

ETEready

</SNAPSHOT>

<TYPE>

system

</TYPE>

<TIME>

2015-02-15 10:05:29.004

</TIME>

<MACHINE>

qas01

</MACHINE>

<VERSION name="CSF">

HEAD

</VERSION>

<VERSION name="Base">

HEAD

</VERSION>

<RESULT>

FAIL

</RESULT>

<FAILURES>

<FAILURE>

<STEP>

7

</STEP>

<REASON>

Unexpected order of execution, see log messages

</REASON>

<LOCATION>

StepRunActionQueueTests

</LOCATION>

</FAILURE>

</FAILURES>



</TEST>

This report format can easily be parsed by an xml parser in any language. It does not have to contain all

of the tags and can repeat tags if necessary, which makes it very flexible for describing the outcome of

tests. Multiple reports can simply be added together and then mined for specific tags.

5.3 RUNNING TESTS USING TAF_INTERACTIVE

The program TAF_interactive can be used to run tests listed in a TAF configure file interactively.

Usage: ./TAF_interactive <TAF configure file path>.

[atst-qas@osllx25 bin]$ ./TAF_interactive TAF.xml


Use /opt/atst as $ATST? (Y/n) >


======================= menu ======================











===================================================

The program reads the TAF configure file and uses <INST_DIR> to set $ATST environment variable.

The program reads the <TESTS> section of the configure file and adds them to its internal list if the

‘index’ attribute is set to the test entry. It is also possible to run a test not listed in the TAF configuration

file by specifying the test file path name or class name.

Below is an example of listing JAVA unit tests.

======================= menu ======================











===================================================

>5

16 Java unit tests

1:atst.qas.tests.csf.REG_CANARY_7_ATCS_627

















16:atst.qas.tests.base.REG_CANARY_7_BASE_135

This is an example of running the Java unit test at index 1.

======================= menu ======================











===================================================

>2

Id? >1

Running Java unit test atst.qas.tests.csf.REG_CANARY_7_ATCS_627

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<TEST>

<DESCR/>

<ID>REG_CANARY_7_ATCS_627</ID>

<CVSID>$Id: REG_CANARY_7_ATCS_627.java,v 1.1 2014/03/12 16:43:45 AlanGreer

Exp $</CVSID>

<VMNAME/>

<TAF_ID>New TAF</TAF_ID>

<SNAPSHOT/>

<TYPE>unit</TYPE>

<TIME>2015-02-16 17:39:55.816</TIME>

<MACHINE>qas01</MACHINE>

<RESULT>PASS</RESULT>

</TEST>

This is an example of running system test by specifying the test script file’s path.

======================= menu ======================













===================================================

>7

Python system test > $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0300.py

Steps (default:all) >

Running Python system test

$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0300.py

[2015-02-17 09:48:13.832] (note:SYS_REQ_4.1_0300.1724.1:) ATST.QA.SYSTEM:

[Step1] Loading a java container SYSREQ410300JAVA1 onto a computer


[Step2] Verifying component is registered


[Step2] The container SYSREQ410300JAVA1 is registered.


[Step3] Loading a java container SYSREQ410300JAVA1 onto a computer


[Step4] Starting up SYSREQ410300JAVA1


[Step5] Adding controller SYS.410300.CTRL.1


[Step5] Controller SYS.410300.CTRL.1 loaded


[Step6] Initializing SYS.410300.CTRL.1


[Step7] Starting up SYS.410300.CTRL.1


[Step8] Verify the language for this controller is JAVA.


[Step8] Controller reports its language as: JAVA.


[Step9] Shutting down SYSREQ410300JAVA1


[Step10] Loading a c++ container SYSREQ410300CPP1 onto a computer


[Step11] Verifying component is registered


[Step11] The container SYSREQ410300CPP1 is registered.


[Step12] Starting up SYSREQ410300CPP1


[Step13] Adding controller SYS.410300.CTRL.2


[Step13] Controller SYS.410300.CTRL.2 loaded


[Step14] Initializing SYS.410300.CTRL.2


[Step15] Starting up SYS.410300.CTRL.2


[Step16] Verify the language for this controller is C++.


[Step16] Controller reports its language as: C++.


[Step17] Shutting down SYSREQ410300CPP1

-----------------------



Test report

-----------------------


<TEST>

<ID>

SYS_REQ_4.1_0300

</ID>

<CVSID>


</CVSID>

<DESCR>

Verify that controllers can be implementable in either Java v1.7 or in

C++11 as specified in requirement 4.1-0300.

</DESCR>

<VMNAME/>

<TAF_ID>

13 Feb

</TAF_ID>

<SNAPSHOT/>

<TYPE>

system

</TYPE>

<TIME>

2015-02-17 09:48:13.826

</TIME>

<MACHINE>

qas01

</MACHINE>

<RESULT>

PASS

</RESULT>

<FAILURES/>

</TEST>

5.4 TESTING AUTOMATION FRAMEWORK CONFIGURATION FILE

The TAF configuration file contains all information required by the TAF to checkout, build and execute

tests on specific virtual machine snapshots. Whenever the ETE starts up a new virtual machine snapshot

instance it will copy the TAF and configuration file onto the snapshot along with the snapshot name. The

TAF will then use the snapshot name as an index to lookup the corresponding entry in the configuration

file, and checkout and build the specified versions of software. The configuration entry may also contain

test tags, which the TAF will use to execute the tests required. It may be possible that an entry contains

no test tags in which case the TAF will not execute any tests. This might be required when the tests are

executing on a different virtual machine snapshot but need to start up containers on other machines. The

table below describes the tags used in the TAF configuration file.

Tag Description

<TAF> An instance of the TAF.

<ID> An ID that will help identify output reports.

<INST_DIR> Specify where the DKIST development tree should be checked out.

For example if /opt is specified the atst module will be checked out

to /opt/atst. If <ATST_DIR> is specified, then the ASDT will be

checked out to <INST_DIR>/<ATST_DIR>.

This tag and the <ATST_DIR> tag are also used to set ATST

environment variable for running tests. The ATST environment



variable will be set to <INST_DIR>/<ATST_DIR>. If

<ATST_DIR> is not specified then it will be set to

<INST_DIR>/atst.

<ATST_DIR> Specify a directory name if ‘atst’ should be checked out to a non-

standard directory. The TAF will use the ‘–d’ cvs option to checkout

the ATST Software Development Tree (ASDT). If this tag is not

specified, the ASDT will be checked out to a standard directory

name. (“atst”)

<VM> An instance of the TAF will execute on this virtual machine

snapshot. The virtual machine name is given.

<REPORT> The location to use for storing test report files.

<SOFTWARE> Define which modules and the version to be checked out. If no

version is specified then the module will not be checked out. To

specify the latest updates in the repository set the version to HEAD.

If no SOFTWARE tag is specified then a warning message will be

printed but execution will continue.

The module name should be set as a tag name and the version as a

text. For example, to checkout trunk version of a module ‘qas’, it

should be specified as <qas>HEAD</qas>

There are five special module names: CSF, BASE, QAS, TCS and

MCS

<CSF> The version of CSF software to checkout.

<BASE> The version of BASE software to checkout.

<TCS> The version of TCS software to checkout.

<QAS> The version of the QAS software to check out.

<MCS> The version of the MCS software to checkout.

<module name> Specify a module name as a tag and the version to be checked out as

text. For example, if <mcs>HEAD</mcs> is set in the

SOFTWARE element, the TAF will checkout the trunk version of

the ‘mcs’ module. Also see 5.2.7checkout_directories.

<INSTALL_APP> Configure whether atst should be checked out and built. If the atst

module is already checked out specify 'no'. When 'no' is set the TAF

will ignore <SOFTWARE>, <SITE_CONFIG>,

<CREATE_DEVEL_ACTION> and <MAKE_ACTION>

<SITE_CONFIG> When building atst, the TAF will copy admin/site.config.template to

admin/site.config and replace values of items in site.config.template

as specified in this section. Specify the name of the item as a tag

name and the value as the text. For example there is an item 'baseDir'

in the site.config.template. To set a value for this item do

<baseDir>/opt/atst</baseDir>

All the items that are not set in the template must be specified here or

the build step of the TAF will fail.

If no SITE_CONFIG tag is found then a warning message will be

printed but execution will continue

<CREATE_DEVEL_ACTION> During the build step, TAF will call admin/createDevel after copying

the site.config file. Specify the parameter which should be called

with this call. It will normally be '--make-all' and this is the default

<MAKE_ACTION> During the build step TAF will call 'make' after the

admin/createDevel call. Specify the parameter which should be



passed with this call. It is normally 'build-all' and if nothing is

specified this is used as the default.

<CHECK_POSTGRESQL> As a default the TAF checks whether a postgresql service is running.

The check can be skipped by setting ‘No’ to this tag. This can be

useful when running a test on a machine that does not host the atst

database.

<POSTGRESQL> To run tests or to call init-db the postgresql service must be running.

The TAF will check if the postgresql service is running by calling

'service <service name> status'. Specify the exact name of the

postgresql service here. This may need the version to abe appended

e.g. postgresql-9.3. If nothing is specified then the service name

'postgresql' is used as the default.

<ICESERVICE> The ICE services have to be running or to create the atst database.

The TAF will call startIceServices unless 'no' is specified here

<INIT-DB> The TAF will call ./admin/createDevel --init-db tp create the atst

database after building the atst. if its already been done then this step

can be skipped by specifying 'no' here.

<TESTS> Define the tests (if any) that are to be executed.

<UNIT> Unit tests to be executed.

<SYSTEM> System tests to be executed.

<JAVA> The location of a Java test script file to execute.

<CPP> The location of a C++ test script file to execute.

<PYTHON> The location of a Python test script file to execute.

Below is an example configuration file for the TAF.

<TAF>

<ID>Canary 7-0</ID>

<INST_DIR>/opt</INST_DIR>

<REPORT>/mnt/hgfs/Reports</REPORT>

<SOFTWARE>

<CSF>Canary_7-0</CSF>

<BASE>Canary_7-0</BASE>

<QAS>HEAD</QAS>

</SOFTWARE>

<INSTALL_APP>no</INSTALL_APP>

<SITE_CONFIG>

<baseDir>$ATST</baseDir>

<ATST_RELEASE_FLAG>Canary_7-0</ATST_RELEASE_FLAG>

<ATST_RUN_ENVIRON>/opt/atst/var</ATST_RUN_ENVIRON>

<ATST_JAVA_HOME>/opt/java</ATST_JAVA_HOME>

<ATST_JLIB_DIRS>none</ATST_JLIB_DIRS>

<USE_CPP>yes</USE_CPP>

<ATST_CLIB_DIRS>/usr/pgsql-9.3/lib/:/usr/local/lib/</ATST_CLIB_DIRS>

<ATST_ARCHIVE_DB_HOST>localhost</ATST_ARCHIVE_DB_HOST>

<ATST_ID_DB_HOST>localhost</ATST_ID_DB_HOST>

<ATST_LOG_DB_HOST>localhost</ATST_LOG_DB_HOST>

<ATST_PROPERTY_DB_HOST>localhost</ATST_PROPERTY_DB_HOST>

<ATST_PARAM_SET_DB_HOST>localhost</ATST_PARAM_SET_DB_HOST>

<ATST_SCRIPT_DB_HOST>localhost</ATST_SCRIPT_DB_HOST>

<ATST_HEADER_DB_HOST>localhost</ATST_HEADER_DB_HOST>

<ATST_ICE_VERSION>3.5.1</ATST_ICE_VERSION>



<ATST_ICE_CONNECTION_HOST>localhost</ATST_ICE_CONNECTION_HOST>

<ATST_ICE_EVENTS_HOST>localhost</ATST_ICE_EVENTS_HOST>

<ATST_ALARM_DB_HOST>localhost</ATST_ALARM_DB_HOST>

</SITE_CONFIG>

<CREATE_DEVEL_ACTION>--make-all</CREATE_DEVEL_ACTION>

<MAKE_ACTION>build_all</MAKE_ACTION>

<POSTGRESQL>postgresql-9.3</POSTGRESQL>

<ICESERVICE>no</ICESERVICE>

<INITDB>no</INITDB>

<TESTS>

<UNIT>

<JAVA>atst.qas.tests.csf.REG_CANARY_7_ATCS_627</JAVA>

<JAVA>atst.qas.tests.csf.REG_CANARY_7_ATCS_632</JAVA>

<CPP>atst/qas/tests/csf/REG_CANARY_7_ATCS_627</CPP>

<CPP>atst/qas/tests/csf/REG_CANARY_7_ATCS_636</CPP>

</UNIT>

<SYSTEM>

<PYTHON>$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py</PYTHON>

<PYTHON>$ATST/src/jython/atst/qas/tests/csf/REG_CANARY_7_ATCS_534.py</PYTHON>

</SYSTEM>

</TESTS>

</TAF>

5.4.1 Testing Automation Framework Configuration File Editing Tool

The TAF configuration file can be edited by the configure_taf program. It can create a new TAF

configuration file, edit an existing file, list current settings in the file or check if the file can be used by

TAF. The usage of configure_taf is:

./configure_taf <operation parameter> <options>

configure_taf reads a file $ATST_TEST/src/python/editor/taf_defaults. The file stores default values

which will be used when creating a new TAF configure file. The file should be edited before using the –

new command.

5.4.1.1 ‘—help’ command

Usage: configure_taf –help

Specify –help parameter to see the help message.

./configure_taf –help

---------------------------TAF Config Editor---------------------------

This python program helps creating/editing TAF config files.

Usage: ./configure_taf <command> <options>

Available commands are:

--new Create a new TAF config file.

--edit Updates a TAF config file.

--list List current setting of a TAF config file

--check Check a TAF config file

--siteconfcopy Update a TAF config file's site conf items from an existing

site.conf file

--siteconf Replace the existing site conf items in the specified TAF

config file with the specified items.



If no item is specified, it will delete all site items.

--addsiteconf Add specified site conf items to the specified TAF config file.

--removesiteconf Remove specified site conf items from the specified TAF config

file.

--copy Copy specified TAF config file to create a new TAF config file.

--help Show this message or specify option to show each command's help

message. (i.e. ./configure_test --help add)

Specify –help with an operation parameter will show a detailed help message of the operation.

configure_taf –help siteconfcopy

./configure_taf --help siteconfcopy

Usage: ./configure_taf --siteconfcopy <TAF config file name> --copyfrom=<site.conf

file>

Update site conf items in a TAF config file from an existing site.conf file.

5.4.1.2 ‘—list’ command

Usage: ./configure_taf –list <TAF configuration file path> <options>

To list the items currently set in a TAF configuration file, specify the parameter –list and the file path.

[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config.xml

<<<< TAF_config.xml START >>>>

ID:New TAF



Softwares:

BASE=HEAD

CSF=HEAD

QAS=HEAD

Unit tests:

JAVA id=1 atst.qas.tests.csf.REG_CANARY_7_ATCS_627



CPP id=1 atst/qas/tests/csf/REG_CANARY_7_ATCS_627



System tests:




Install application:No

Site config settings:

ATST_ICE_VERSION = 3.5.1

ATST_ID_DB_HOST = localhost

ATST_LOG_DB_HOST = localhost

TST_HEADER_DB_HOST = localhost

USE_CPP = yes

baseDir = $ATST

ATST_BULK_DATA_DB_HOST = localhost

ATST_PARAM_SET_DB_HOST = localhost

ATST_JLIB_DIRS = none



ATST_RELEASE_FLAG = trunk

ATST_ALARM_DB_HOST = localhost

ATST_PROPERTY_DB_HOST = localhost

ATST_RUN_ENVIRON = /opt/atst/var

ATST_ICE_EVENTS_HOST = localhost

TST_SCRIPT_DB_HOST = localhost

ATST_JAVA_HOME = /opt/java8

ATST_ICE_CONNECTION_HOST = localhost

ATST_CLIB_DIRS = /usr/pgsql-9.3/lib/:/usr/local/lib/

ATST_ARCHIVE_DB_HOST = localhost

createDevel action:--make-all

make action:build_all


init-db:Yes


<<<< TAF_config.xml END >>>>

Items to be listed can be specified. Item options are: id, installdir, reportdir, software, unittests,

systemtests, appinstall, siteconfig, cdaction, maction, postgresql, initdb and ice.

[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config.xml id systemtests unittests


ID:New TAF

Unit tests:







System tests:





5.4.1.3 ‘—new’ command

Usage: ./configure_taf –new <TAF configuration file path> <options>

To create a new TAF configuration file, call configure_taf with parameter ‘–new’. The TAF configure file

path and –id option is required. Default values for each item are stored in the

$ATST_TEST/src/python/editor/taf_defaults file . When creating a new TAF configure file, the default

values are used for items that are not specified in the options. Specify –nodefaults to not use any default

values.

List of options available to –new are:

--id(Required) The ID for the new TAF config file.

--install_dir Specify where the ATST will be checked out/installed.

(i.e. /opt)

--report_dir Specify where test report should be stored.

(i.e. /opt/Reports).

--software Specify software and its versions to be installed.

(i.e. csf=Canary_7-0,base=Canary_7-0)

--create_devel_action Specify createDevel action. (i.e. --make-all)

--make_action Specify make action. (i.e. build_all)

--install_app Specify whether to checkout and install atst.(yes or no)

--java_unit_test Specify JAVA unit tests, separated by ','.

(i.e. atst.qas.tests.csf.REG_CANARY_7_ATCS_627)



--java_unit_id When specifying JAVA unit tests with --java_unit_test option,

ids for each test can be specified using this option.

--cpp_unit_test Specify CPP unit tests, separated by ','.

(i.e. atst/qas/tests/csf/REG_CANARY_7_ATCS_627)

--cpp_unit_id When specifying CPP unit tests with --cpp_unit_test option,

ids for each test can be specified using this option.

--python_system_test Specify python system tests, separated by ','.

(i.e. '$ATST'/src/jython/atst/qas/tests/csf/SYS_0101.py)

--python_system_id When specifying Python system tests with --python_system_test

option,ids for each test can be specified using this option.

--postgresql Specify postgresql program name. (i.e. posgresql-9.3)

--initdb Specify whether to initialize database. (yes or no)

--iceservice Specify whether to call startIceServices. (yes or no)

--nodefaults Specify this option to not use any default values.

For the example below, the taf_default file is set as follows.

[atst-qas@qas01 bin]$ more $ATST_TEST/src/python/editor/taf_defaults

INST_DIR=/opt

REPORT=/mnt/hgfs/Reports

CREATE_DEVEL_ACTION=--make-all

MAKE_ACTION=build_all

POSTGRESQL=postgresql-9.3

INITDB=Yes

ICESERVICE=Yes

INSTALL_APP=No

#Software defaults should be placed between 'SOFTWARES_START' and 'SOFTWARES_END'

#It should be specified as <Software name>=<Version>

SOFTWARES_START

CSF=HEAD

BASE=HEAD

QAS=HEAD

SOFTWARES_END

#Site config default items should be placed between 'SITE_CONFIG_START' and

'SITE_CONFIG_END'

SITE_CONFIG_START

baseDir=$ATST

ATST_RELEASE_FLAG=trunk

ATST_RUN_ENVIRON=/opt/atst/var

ATST_JAVA_HOME=/opt/java8

ATST_JLIB_DIRS=none

USE_CPP=yes

ATST_CLIB_DIRS=/usr/pgsql-9.3/lib/:/usr/local/lib/

ATST_ARCHIVE_DB_HOST=localhost

ATST_ID_DB_HOST=localhost

ATST_LOG_DB_HOST=localhost

ATST_PROPERTY_DB_HOST=localhost

ATST_PARAM_SET_DB_HOST=localhost

TST_SCRIPT_DB_HOST=localhost

TST_HEADER_DB_HOST=localhost

ATST_ICE_VERSION=3.5.1

ATST_ICE_CONNECTION_HOST=localhost

ATST_ICE_EVENTS_HOST=localhost

ATST_ALARM_DB_HOST=localhost

ATST_BULK_DATA_DB_HOST=localhost

SITE_CONFIG_END

#System tests should be placed between 'SYSTEM_TESTS_START' and 'SYSTEM_TESTS_END'

#System test should be specified as <TEST path>, <id(optional)>

SYSTEM_TESTS_START



$ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py,1



SYSTEM_TESTS_END

#Java Unit tests should be placed between 'JAVA_TESTS_START' and 'JAVA_TESTS_END'

#Java Unit test should be specified as <TEST class path>, <id(optional)>

JAVA_TESTS_START

atst.qas.tests.csf.REG_CANARY_7_ATCS_627,1



JAVA_TESTS_END

#Cpp Unit tests should be placed between 'CPP_TESTS_START' and 'CPP_TESTS_END'

#Cpp Unit test should be specified as <TEST class path>, <id(optional)>

CPP_TESTS_START

atst/qas/tests/csf/REG_CANARY_7_ATCS_627,1



CPP_TESTS_END

The example below shows how to create a new TAF configuration file using default values. Only the –id

option is specified but other tags are added using default values.

[atst-qas@qas01 bin]$ ./configure_taf --new TAF_config.xml --id="New TAF"

New TAF configure file (TAF_config.xml) created.

[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config.xml


ID:New TAF



Softwares:

BASE=HEAD

CSF=HEAD

QAS=HEAD

Unit tests:







System tests:




Install application:No





TST_HEADER_DB_HOST = localhost

USE_CPP = yes

baseDir = $ATST











TST_SCRIPT_DB_HOST = localhost








init-db:Yes



An example of creating a new configure file without using default values.

[atst-qas@qas01 bin]$ ./configure_taf --new TAF_config_2.xml --id="New TAF" --install_

dir="/opt" --report_dir=/mnt/hgfs/Reports --software=CSF=Canary_8,BASE=Canary_8,QAS=

HEAD --nodefaults

New TAF configure file (TAF_config_2.xml) created.

[atst-qas@qas01 bin]$ ./configure_taf --list TAF_config_2.xml

<<<< TAF_config_2.xml START >>>>

ID:New TAF



Softwares:

CSF=Canary_8

BASE=Canary_8

QAS=HEAD

<<<< TAF_config_2.xml END >>>>

5.4.1.4 ‘—edit’ command

Usage: ./configure_taf –edit <TAF configure file> <options>

To edit an existing TAF configuration file, call configure_taf with parameter ‘–edit’ and the TAF

configure file’s path followed by options. The options available to –edit option are:

--id The ID of the TAF config file.

--install_dir Specify where the ATST will be checked out/installed.

(i.e. /opt).

--report_dir Specify where test report should be stored.

(i.e. /opt/Reports).

--software Replace the software entry with the specified values.

(i.e. csf=Canary_7-0,base=Canary_7-0)

--add_software Add specified software entry to the existing softwares.

(i.e. csf=Canary_7-0,base=Canary_7-0) If the specified

software already exists, it will be overwitten.

--remove_software Remove the software entry from the existing softwares.

(i.e. csf,base)

--create_devel_action Specify createDevel action. (i.e. --make-all)

--make_action Specify make action. (i.e. build_all)

--install_app Specify whether to checkout and install atst.

(yes or no)

--java_unit_test Replace the JAVA unit tests entry.


--add_java_unit_test Add JAVA unit tests entry.


--rem_java_unit_test Remove specified JAVA unit tests.


--cpp_unit_test Replace the CPP unit tests.




--add_cpp_unit_test Add CPP unit tests, separated by ','.


--rem_cpp_unit_test Remove specify CPP unit tests, separated by ','.


--python_system_test Replace the python system tests, separated by ','.

(i.e. '$ATST'/src/jython/atst/qas/tests/csf/SYS0101.py)

--add_python_system_test Add python system tests, separated by ','.

--rem_python_system_test Remove specified python system tests, separated by ','.

--postgresql Specify postgresql program name. (i.e. posgresql-9.3)

--initdb Specify whether to initialize database. (yes or no)

--iceservice Specify whether to call startIceServices. (yes or no)

This is an example of updating ‘software’ and ‘iceservice’:



ID:New TAF



Softwares:

CSF=Canary_8

BASE=Canary_8

QAS=HEAD


[atst-qas@qas01 bin]$ ./configure_taf --edit TAF_config_2.xml --software=CSF=HEAD,BASE

=HEAD,QAS=HEAD --iceservice=Yes

TAF config file (TAF_config_2.xml) is saved.



ID:New TAF



Softwares:

CSF=HEAD

BASE=HEAD

QAS=HEAD

Start ICE service:yes


5.4.1.5 ‘—check’ command

Usage: ./configure_taf –check <TAF configuration file path>

Specify the –check parameter to check if a TAF configuration file will work with TAF program. It will

check if any required tags are missing and that the content of the file is well formed.

./configure_taf --check ../../Configs/TAF_config_Canary8_example.xml

TAF config file OK.

./configure_taf --check ../../Configs/TAF_config_2.xml

TAF Config file chek FAILED - Exception occured while reading the TAF configuration

file.

<class 'xml.parsers.expat.ExpatError'>

not well-formed (invalid token): line 27, column 2



./configure_taf --check TAF_config_3.xml

Warning:

ID tag not found in the configuration file

5.4.1.6 Command to edit SITE_CONFIG items

5.4.1.6.1 –siteconfcopy command

Usage: ./configure_taf –siteconfcopy <TAF configuration file path> --copyfrom=<site.config file path>

SITE_CONFIG items can be copied into a TAF configure file from an existing site.config file. Specify –

siteconfcopy parameter, the TAF configuration file path and the site.config file’s path.

./configure_taf --siteconfcopy ../../Configs/taf_config.xml --

copyfrom=../../Configs/site.config

TAF config file (../../Configs/taf_config.xml) is saved.

./configure_taf --list ../../Configs/taf_config.xml

<<<< ../../Configs/taf_config.xml START >>>>

ID:New TAF



Softwares:

CSF=HEAD

BASE=HEAD

QAS=HEAD



baseDir = $ATST


ATST_CPP_CPP_BIN = /opt/rh/devtoolset-2/root/usr/bin/g++



ATST_ICE_LIB_SUFFIX = 35


ATST_JAVAC_FLAGS = "-Xlint:all,-path,-serial"

ATST_ICE_JAR =

/usr/share/java/Ice.jar:/usr/share/java/IceStorm.jar:/usr/share/java/IceGrid.jar:/usr/

share/java/Glacier2.jar



ATST_ICE_EVENTS_PORT = 12000

ATST_JAVA_DOXYGEN = no

ATST_JAVA_ICE_LIB_PATH = /usr/share/java

ATST_TARGET_ARCH = $(uname -m)


ATST_ICE_CONNECTION_PORT = 11000

USE_ICE = yes

ATST_CONCURRENT_CMAKES = 0

ATST_CCINC_DIRS = /usr/local/include/zdb





ATST_JAVA_FLAGS = "-XX:+UseConcMarkSweepGC -Xmx4G"

ATST_PACKAGE_LIST = "base"

ATST_SCRIPT_DB_HOST = localhost



ATST_DB_JAR = /usr/share/java/db-5.3.21.jar

ATST_EXPERIMENT_DB_HOST = none

USE_JAVA = yes


ATST_HEADER_DB_HOST = localhost

USE_CPP = yes

ATST_CC_BIN = /opt/rh/devtoolset-2/root/usr/bin/gcc

ATST_JNI_PACKAGES = "none"

ATST_ICE_ICEBOX_PORT = 11998

ATST_ICE_HOME = /usr




<<<< ../../Configs/taf_config.xml END >>>>

5.4.1.6.2 –siteconf command

Usage: ./configure_taf –siteconf <TAF configuration file path> <SITE_CONFIG items>

Specify –siteconf parameter to replace existing SITE_CONFIG items in a TAF configure file.

SITE_CONFIG items should be specified in the format <name>=<value>.

./configure_taf --siteconf ../../Configs/taf_config.xml

ATST_ICE_CONNECTION_HOST=localhost baseDir='$ATST' USE_CPP=no




ID:New TAF



Softwares:

CSF=HEAD

BASE=HEAD

QAS=HEAD


USE_CPP = no

baseDir = $ATST




5.4.1.6.3 – addsiteconf command

Usage: ./configure_taf –addsiteconf <TAF configuration file path> <SITE_CONFIG items>

Specify the –addsiteconf parameter to add more SITE_CONFIG items to a TAF configure file. If the

specified SITE_CONFIG item already exists in the TAF configure file, the value of the item will be

updated.

./configure_taf --addsiteconf ../../Configs/taf_config.xml USE_CPP=yes ATST_ICE_CONNEC

TION_PORT=11000 ATST_JAVA_HOME=/opt/java8






ID:New TAF



Softwares:

CSF=HEAD

BASE=HEAD

QAS=HEAD


USE_CPP = yes

baseDir = $ATST


ATST_ICE_CONNECTION_PORT = 11000




5.4.1.6.4 – removesiteconf command

Usage: ./configure_taf –removesiteconf <TAF configuration file path> <SITE_CONFIG item names>

Specify the –removesiteconf parameter to remove SITE_CONFIG items from a TAF configure file.

./configure_taf --removesiteconf ../../Configs/taf_config.xml USE_CPP ATST_ICE_CONNEC

TION_PORT




ID:New TAF



Softwares:

CSF=HEAD

BASE=HEAD

QAS=HEAD


baseDir = $ATST





5.4.1.7 –copy command

Usage: ./configure_taf –copy <Existing TAF configuration file path> --newfile=<New TAF configuration

file path> --id=<new ID>

Specify the –copy parameter to create a new TAF configure file by copying an existing TAF configure

file. Specify the new file’s path with the –newfile option and the --id option to set a new ID in the new

file.

./configure_taf --copy ../../Configs/taf_config.xml --newfile=../../Configs/taf_config

_new.xml --id='New TAF 2'

TAF configure file (../../Configs/taf_config_new.xml) created.



./configure_taf --list ../../Configs/taf_config_new.xml

<<<< ../../Configs/taf_config_new.xml START >>>>

ID:New TAF 2



Softwares:

CSF=HEAD

BASE=HEAD

QAS=HEAD


baseDir = $ATST




<<<< ../../Configs/taf_config_new.xml END >>>>

5.4.1.8 Interactive configuration tool configure_tests

The configure_tests program can be used to edit TAF configuration files interactively. Below is an

example of creating and editing a new TAF configuration file. Pressing Ctrl+D will cancel current

editing process and will go back to a previous menu. When editing, the changes that are made to the TAF

configuration will not be saved to the file until ‘Save’ is selected. ‘*’ will appear next to the file name to

indicate TAF configuration has unsaved changes. This program also makes use of the default file

$ATST_TEST/src/python/editor/taf_defaults

./configure_tests

--------------configure_tests--------------

(Ctrl+D can be used to cancel current editing process)

Create New file or Edit existing file (New/Edit)? >n

Which config file to create? (T)AF or (E)TE)? >t

Creating new TAF config file.

Please specify the new TAF config file path >../../new_taf_config

note:Extension '.xml' is added to the specified file name

Creating new TAF config file.

ID >Created by configure_tests

Install directory >/opt

Report file's directory >/mnt/hgfs/Reports

New file <../../new_taf_config.xml> created

Use Edit menu to add more items.

-------Editing TAF config file < new_taf_config.xml >-------

1:Edit 2:CHeck 3:List 4:ChangeFile

5:Save 6:Help 7:Quit

>3

<<<< ../../new_taf_config.xml START >>>>

ID:Created by configure_tests



<<<< ../../new_taf_config.xml END >>>>




>ch

TAF config file OK.






>1

---Edit items <new_taf_config.xml> ------

Available items to edit are

1: id 2: install_dir 3: report_dir 4: software

5: cdaction 6: maction 7: appinstall 8: java_unit_test

9: cpp_unit_test 10: python_system_test

11: postgresql 12: init-db 13: iceservice 14: siteconf

Item to update (Just hit enter to quit editing itmes.) > 4

Current Software:''

Software: Add/update, Delete, Replace, List? >a

Software name > CSF

version > Canary_8

Softwares updated.

Softwares to add > BASE

version > Canary_8

Softwares updated.

Softwares to add > QAS

version > HEAD

Softwares updated.

Softwares to add >

Software: Add/update, Delete, Replace, List? >l

Softwares:

CSF=Canary_8

BASE=Canary_8

QAS=HEAD

Software: Add/update, Delete, Replace, List? >

---Edit items <new_taf_config.xml*> ------

Available items to edit are

1: id 2: install_dir 3: report_dir 4: software

5: cdaction 6: maction 7: appinstall 8: java_unit_test

9: cpp_unit_test 10: python_system_test

11: postgresql 12: init-db 13: iceservice 14: siteconf

Item to update (Just hit enter to quit editing itmes.) >

End editing.

-------Editing TAF config file < new_taf_config.xml* >-------



>l

<<<< ../../new_taf_config.xml START >>>>

ID:Created by configure_tests



Softwares:

CSF=Canary_8

BASE=Canary_8

QAS=HEAD

<<<< ../../new_taf_config.xml END >>>>

-------Editing TAF config file < new_taf_config.xml* >-------



>s

TAF config file (../../new_taf_config.xml) is saved.






>cf

Create New file or Edit existing file (New/Edit)? >e

Please specify the config file name to edit. >

../../Configs/TAF_config_Canary8_example.xml

-------Editing TAF config file < TAF_config_Canary8_example.xml >-------



>3

<<<< ../../Configs/TAF_config_Canary8_example.xml START >>>>

ID:Canary 8 Checkout and tests



Softwares:

CSF=Canary_8

BASE=Canary_8

QAS=HEAD

Unit tests:

JAVA atst.qas.tests.csf.REG_CANARY_8_ATCS_791


CPP atst/qas/tests/csf/REG_CANARY_8_ATCS_791


System tests:

PYTHON $ATST/src/jython/atst/qas/tests/csf/SYS_REQ_4.1_0101.py



Install application:yes


baseDir = $ATST

ATST_RELEASE_FLAG = Canary_8




USE_CPP = yes

















init-db:yes


<<<< ../../Configs/TAF_config_Canary8_example.xml END >>>>




>2

TAF config file OK.






>7



6. E2E TEST EXECUTOR

The E2E Test Executor (ETE) is responsible for scheduling automated test runs, setting up and running

the required virtual machines and final processing of the report. The ETE is formed as a set of

scripts/programs that are executed as a cron job and can be easily configured from the command line. A

full explanation of the configuration options is provided in section 6.3. The ETE can be configured to

start up a test run at any pre-defined time on any available virtual machines. When the specified time is

reached the ETE will wake up, start the necessary virtual machines and then copy the TAF scripts and

configuration files to the virtual machines. After the copies have completed the ETE will execute TAF

scripts on the virtual machines and wait for the completion of all tests. Once the tests have been

completed the ETE will shut down the virtual machines and then perform the appropriate notifications of

the test results. The ETE will be responsible for ensuring only one test run is performed at any one time

on one virtual machine, but will not stop multiple virtual machines from being active and running

separate test runs at any one time. From the command line it will be possible for an operator to invoke

the ETE to start a test run immediately.

For situations where real hardware might be used for testing the ETE would not be required, instead

configured instances of the TAF would be manually executed on physical machines. The result would be

a full set of tests run with real hardware and reports stored in the configured location.

6.1 END TO END SIMULATOR

The end-to-end simulator (E2E) is described as the software and hardware system used to integrate and

test all software component, assemblies, and systems that are part of the DKIST control system. The E2E

consists of three servers located in the DKIST Tucson office, and the software QA will form a set of

virtual machine snapshots along with the tools necessary to execute on one of those machines. For

network performance tests it may be necessary to expand and have one extra virtual machine snapshot on

a separate physical machine, separated by a physical network.

The E2E does not contain any hardware required for the subsystems, and so each subsystem must be

executed in a fully simulated state. Any special procedures necessary to execute the subsystems in a

simulated state must be possible through the use of scripts so that the TAF can automate this setting of

subsystems into their simulated state.

6.2 RUNNING THE ETE

The following steps describe how to run the ETE on a machine that has VMWare installed and at least

one prepared Virtual Machine.

1. Checkout the atst_test repository onto the host machine.

export CVSROOT=:pserver:<username>@maunder.tuc.noao.edu:/home/atst/src/qas

cvs login


2. Build and install the atst_test source files.

export ATST_TEST=/opt/atst_test

cd $ATST_TEST





3. Make sure that there is a shared folder setup between host and the virtual machine that will be

used for executing the tests. The shared folder is used to store the log files from the test

execution, which preserves any log files for review after the tests have completed and the virtual

machine has been shut down.

4. Optionally, login to the virtual machine, and then cvs login to csf, base and qas with the id “atst-

qas”.

5. Create a TAF configuration file on the host machine. The TAF configuration file will be copied

onto the virtual machine by the ETE application. An example of a TAF configuration file is in

$ATST_TEST/src/python/taf/TAF_config_example.xml

Note that test script files should exist in the virtual machine. (Normally they should exist in cvs

so that they will be checked out). Also regarding the SOFTWARE, if no version is specified for a

module, then that module won’t be checked out. For example,

<SOFTWARE>

<CSF>Canary_7</CSF>

<BASE></BASE>

<TCS>HEAD</TCS>

<QAS>HEAD</QAS>

</SOFTWARE>

In this case, cvs checkout won’t be called for ‘base’.

Also, if <INSTALL_APP>no</INSTALL_APP> is specified, then nothing will be checked out

or built. This is useful for running tests on an already installed system ,where there is no

requirement to start from a completely fresh checkout on each execution of the TAF.

There are tools to edit TAF configuration file. See 5.4.1 for more detail.

6. Create an ETE configuration file on the host machine. An example is in

$ATST_TEST/src/python/ete/examples/ETE_config_example.xml

<REPORT_DIR_PATH_ON_VM> Specify the path of the shared folder on the virtual machine.

<REPORT_DIR_PATH_ON_HOST> Specify the path of the shared folder on the host.

There are tools to edit TAF configuration file. See 6.3.2 for more detail.

7. Run the following command to start the ETE.

./ETE --eteconfig="path to the ETE config file" --vmuser=<vm user name> --

vmpassword=<vm user password>

If cvslogin hasn’t been done at step 4, then the --cvsuser and --cvspassword need to be specified.

By default it runs the vm without GUI. Specify --vmwithgui to get a vm gui.

Also by default it shutdowns the virtual machine after the test, which is quite time consuming.

We can set --noshutdown to leave the virtual machine as it is (running) or --suspend to just

suspend it. Please note that ETE will search for a test instance to be executed at the current time.



To ignore the current time and execute a specific test instance the --testtime option should be

specified.

The options for ETE are:

--eteconfig= Set the ETE config file path. (Required)

--vmuser= Set the virtual machine user. (Required)

--vmpassword= Set the virtual machine user's password. (Required)

--cvsuser= Set CVS user for checking out atst software. (Optional)

--cvspassword= Set CVS password for checking out atst software. (Optional)

--emailpgm= Set which email program to use. 'mail'(using 'localhost' SMTP and

current user ID) or 'gmail'(requires --emailuser and

--emailpassword options.). (Optional)

--emailuser= Set Email user for sending final report only if ‘gmail’ is

selected for --emailpgm. (Optional)

--emailpassword= Set the mail user's password only if ‘gmail’ is selected for

--emailpgm. (Optional)

--noshutdown Specify this if the virtual machine shouldn't be shutdown after

the test.(Optional)

--suspend Specify this to 'suspend' the virtual machine rather than

'shutdown' after the test. This is ignored if --noshutdown is

specified. (Optional)

--vmwithgui Specify this to run the VM with GUI.(Optional)

--ignoretime Specify this to run a test instances whose 'time' is specified as

'NOW'. (Optional)

--testtime Specify this to run a test instances of the specified

time. (Optional)

--ignoretime and --testtime options can't be specified together

--help Show help message

Very briefly, how ETE works is:

ETE starts the virtual machine.

ETE copies TAF source code and TAF configuration file into the virtual machine’s home directory.

ETE runs the TAF.sh which has just been copied to the virtual machine.

ETE waits for the signal that TAF has been finished. (There is a lock file in the /tmp folder of the

virtual machine and ETE reads it periodically.)

TAF on the virtual machine reads the configuration file which was copied by ETE.

TAF checks out the modules specified in the config file (unless specified not to).

TAF updates site.config as specified in the config file.(unless specified not to).

TAF builds atst.(unless specified not to).

TAF starts postgresql

TAF starts ICE services (unless specified not to)

TAF does init_db

TAF runs system tests and unit tests.

If there are more VMs to run, the ETE runs the vm and then runs TAF.

When there are no more VMs to run, ETE shuts down all the virtual machines, writes the final report and,

optionally, emails the report.



6.3 CONFIGURATION OF THE ETE

The ETE executes using a set of configuration files. The configuration files are text based xml files that

can be opened by any editor, but a configuration tool is provided to ensure they are correctly formatted.

6.3.1 ETE Configuration File

The ETE configuration file contains all information required for the ETE to be able to start virtual

machine snapshots and to email the results to interested parties. Each instance of ETE execution is

indexed by the time at which the instance should begin execution. The instance contains the names of the

virtual machine snapshots to start (this can be any number of virtual machines) and the email addresses to

send the generated report files to. The number of execution instances is limited only by the requirement

that each instance must begin at a different time. Each instance is indexed by the time and so only one

can exist for each (HH:MM). The ETE will not attempt to start virtual machine snapshots if they are

already running, and will instead report this as an error. The table below describes the tags used in the

ETE configuration file.

Tag Description

<instance> Test run instance. Requires ‘time’ attribute to be set in 24 hour

format HH:MM in UTC to indicate what time the test should

be run. ‘NOW’ can also be specified here to be used with --

ignoretime option.

<EXECUTION> Execution instance. It contains virtual machine instances to run

in the test run instance.

<VM> Virtual Machine instance. A set of information for running

TAF on a virtual machine are specified here. More than one

virtual machine instance can be specified in an execution

instance. Tags which must be specified here are <VM_FILE>,

<TAF_CONFIG>, <REPORT_DIR_PATH_ON_VM>,

<REPORT_DIR_PATH_ON_HOST>,

<SNAPSHOT> and <RUN_ORDER>.

<VM_FILE> Virtual machine file name.

<TAF_CONFIG> Path of the TAF config file to be used on the virtual machine.

<REPORT_DIR_PATH_ON_VM> Path of the shared folder which is set up between the virtual

machine and the host machine to store test reports, on the

virtual machine

<REPORT_DIR_PATH_ON_HOST> Path of the shared folder which is set up between the virtual

machine and the host machine to store test reports, on the host

machine.

<SNAPSHOT> Name of the snapshot of the virtual machine to revert to. It is

possible to leave it empty. In this case, the ETE will not

change the virtual machine state.

<RUN_ORDER> The order of the VM instance to be executed by ETE.

<SYSTEM_TESTS> Optional tag.

Specify the Ids of the system tests which should be executed

by TAF. Multiple Ids can be specified by separating them by

‘,’. ‘-‘ can be used to specify the multiple tests between two

Ids. For example, specifying ‘1-3’ will execute test Ids 1,2, and

3. Specify ‘all’ to run all the system tests.

Steps of each test to be executed can be specified as ‘[1,2,...]’

after each test Ids. For example, to run steps 1,2,3 of a test of



Id 1, then it should be specified as ‘1[1-3]’.

Note: If none of tags <SYSTEM_TESTS>,

<JAVA_UNIT_TESTS> and <CPP_UNIT_TESTS> is present

then TAF will run all the tests specified in the TAF configure

file.

<JAVA_UNIT_TESTS> Optional tag.

Specify the Ids of the Java unit tests which should be executed








file.

<CPP_UNIT_TESTS> Optional tag.

Specify the Ids of the CPP unit tests which should be executed








file.

<EMAIL_ADDRESS> Email address to send test report to.

<FINAL_REPORT_DIR> The path of the directory where a final report should be stored.

Below is an example configuration file for the ETE. This ETE configure file, at 10:00 UTC ETE will

start up the virtual machine ATST_READY.vmx and run tests specified in the TAF_config_HEAD.xml.

After it completes, it will then startup the virtual machine ATST_READY2.vmx and run the tests

specified in the TAF_config_HEAD2.xml. The ETE will then shutdown both virtual machines and create

the final report in the folder specified in the FINAL_REPORT_DIR tag and email the report to the

address specified in the EMAIL_ADDRESS tag. At 12:40 UTC ETE will start up the virtual machine

ATST_READY.vmx and run the tests specified in the TAF_config_HEAD.xml. When it finishes it will

shut down the virtual machine and create a final report in the folder specified by the

FINAL_REPORT_DIR tag.

<ETE>

<instance time="10:00">

<EXECUTION>

<VM>

<VM_FILE>/drive/VMWareMachines/ATST_READY/ATST_READY.vmx</VM_FILE>

<TAF_CONFIG>/drive/ATST_TEST/Configs/TAF_config_HEAD.xml</TAF_CONFIG>

<REPORT_DIR_PATH_ON_VM>/mnt/hgfs/Reports</REPORT_DIR_PATH_ON_VM>

<REPORT_DIR_PATH_ON_HOST>/drive/ATST_TEST/Reports</REPORT_DIR_PATH_ON_HOST>

<SNAPSHOT>Ready</SNAPSHOT>

<RUN_ORDER>1</RUN_ORDER>

</VM>

<VM>

<VM_FILE>/drive/VMWareMachines/ATST_READY/ATST_READY2.vmx</VM_FILE>

<TAF_CONFIG>/drive/ATST_TEST/Configs/TAF_config_HEAD2.xml</TAF_CONFIG>







</VM>

</EXECUTION>

<EMAIL_ADDRESS>Aya <[email protected]> </EMAIL_ADDRESS>

<FINAL_REPORT_DIR>/drive/ATST_TEST/FinalReport</FINAL_REPORT_DIR>

</instance>


<EXECUTION>

<VM>


<VM_FILE>/drive/VMWareMachines/ATST_READY/ATST_READY.vmx</VM_FILE>

<TAF_CONFIG>/drive/ATST_TEST/Configs/TAF_config_Canary8.xml</TAF_CONFIG>




</VM>

</EXECUTION>

<FINAL_REPORT_DIR>/drive/ATST_TEST/FinalReport</FINAL_REPORT_DIR>

</instance>

</ETE>

6.3.2 Configuration Tool

The Configuration tool configure_ete program can be used to edit ETE configuration files to avoid the

necessity of altering the xml files directly. The tool provides abilities to edit, check and list the contents of

an ETE configuration file. The usage of the tool is:

./configure_ete <command parameter> <options>

The configure_ete command uses default values for items that are not specified in the options when

creating a new configure file or editing, whenever possible. The default values are stored in a file

$ATST_TEST/src/python/editor/ete_defaults. The file needs to be updated before running configure_ete.


--new Create a new empty ETE config file.

--add Add a new test instance.

--copy Copy an ETE config file to create a new ETE config file.

--edit Update items.

--del Delete a test instance.

--addvm Add a new vm instance to a test instance.

--deletevm Delete vm instances from a test instance.

--list List current setting of a ETE config file

--check Check an ETE config file

--changetime Change the execution time of a test instance.



6.3.2.1 ‘--list’ command

Usage: ./configure_tests –list <ETE configure file path>

To list all instances of ETE execution use the --list parameter followed by the path of the ETE

configuration file.



[qas@osllx25 bin]$ ./configure_ete --list ../../Configs/ETE_config.xml

Listing ETE config file entries.

Time : 10:00


Email: Aya <[email protected]>

Run order: 1


Snapshot name: Ready

TAF config file: /drive/ATST_TEST/Configs/TAF_config_HEAD.xml



6.3.2.2 ‘--new’ command

./configure_ete –new <ETE Configure file path>

To create a new empty ETE configuration file, specify the –new parameter with the file path. If the

specified file path doesn’t have any extension, .xml will be added.

./configure_ete --new ../../Configs/ete_conf

note: Extension .xml is added to the file name.


New ETE config file (../../Configs/ete_conf.xml) created.

6.3.2.3 ‘--add’ command

Usage: ./configure_ete –new <ETE configuration file path> <options>

To add a new execution instance to the ETE, the --add parameter is used.

The instance is indexed by time. The final report directory and emails can also be specified. Virtual

machine instances can be specified with –vm(n) option. The number ‘n’ represents the run order of the

virtual machine instance. Specify items VM_FILE, TAF_CONFIG, REPORT_DIR_PATH_ON_VM,

REPORT_DIR_PATH_ON_HOST, SNAPSHOT separated by ',' without spaces. Specify --

systemtests_vm(n) to set the SYSTEM_TESTS tag, --javaunittests_vm(n) to set the

JAVA_UNIT_TESTS tag and –cppunittests_vm(n) to set the CPP_UNIT_TESTS tag.

The only required option for the add command is –time. For any of the options that are not specified, the

default values will be used. To avoid using the default values, specify –nodefaults.

The list of available options for the –add command are:

--time(required) The time when the test should run. The format is HH:MM.

--frd The path to the directory where final report will be

stored.

--emails Email addresses. If specified, the final report will be

emailed. Multiple address can be specified separated with

','.

--vm(n) Virtual machine settings. Specify items <VM_FILE>,

<TAF_CONFIG>,<REPORT_DIR_PATH_ON_VM>,

<REPORT_DIR_PATH_ON_HOST>, <SNAPSHOT> separated by ','

without spaces. Specify--vm with number of order. For

example, if adding <VM> with order number of 2, the option

name is --vm2.



--nodefaults Specify this option to not use any default values. If this

option is not specified, default values will be used for

items that are not specified.

--systemtests_vm(n) Specify the IDs of system tests which should be run by TAF.

The ‘n’ represents the order of the VM instance.

--javaunittests_vm(n) Specify the IDs of Java unit tests which should be run by

TAF. The ‘n’ represents the order of the VM instance.

--cppunittests_vm(n) Specify the IDs of CPP unit tests which should be run by


For the example below, the ete_default file is set as follows.

VM_FILE=/drive/VMWareWorkstation/ATST_READY/ATST_READY.xml

TAF_CONFIG=/drive/ATST_TEST/Configs/TAF_config_HEAD.xml


REPORT_DIR_PATH_ON_HOST=/drive/ATST_TEST/Reports

FINAL_REPORT_DIR=/drive/ATST_TEST/FinalReport

[email protected]

SNAPSHOT=Ready

SYSTEM_TESTS=all

JAVA_UNIT_TESTS=1-5

CPP_UNIT_TESTS=all

This example adds a new test instance at 12:00 by specifying only the –time option. It demonstrates that

VM instance with order number 1 is added using default values.

[aya@osllx21 bin]$ ./configure_ete --add ETE.xml --time=12:00


VM order 1 with default values is added.

Default email addresses are added.

Default final report directory is used.

ETE config file (ETE.xml) is saved.

-----ETE configure file - ETE.xml -----

Time : 12:00


Email: [email protected]

Run order: 1

VM file: /drive/VMWareWorkstation/ATST_READY/ATST_READY.xml



System tests: all

Java unit tests: 1-5

CPP unit tests: all



---------------end-------------------

When –nodefaults option is specified, no items other than those specified in the options will be added.

[aya@osllx21 bin]$ ./configure_ete --add ETE.xml --time=14:00 --frd=$ATST_TEST/

FinalReport [email protected] --nodefaults




Time : 12:00





Run order: 1




System tests: all


CPP unit tests: all



Time : 14:00

Final report directory: /home/aya/ATST_TEST/atst_test/FinalReport


---------------end-------------------

6.3.2.4 ‘--del’ command

Usage: configure_ete --del < ETE configure file path > --time <HH:MM>

To remove a previously defined test instance from the ETE, use the --del parameter. The instance to

remove needs to be indexed by time.

[aya@osllx21 bin]$ ./configure_ete --del ETE.xml --time=12:00

Deleting a test instance...

Test instance (12:00) removed.



Time : 14:00



---------------end-------------------

6.3.2.5 ‘--addvm’ command

Usage: ./configure_ete --addvm <ETE configure file path> --time=<time> --order=<VM order>

<options>...

To add a new VM instance to a test instance, the --addvm parameter is used.

--time option is required to specify which test instance the new

VM will be added to. The order of the virtual machine is

specified by the –order option. Other items of the VM

instance can be specified using the options listed below.

Similar to the –add command, the default values will be

used unless –nodefaults option is specified.

Available options for '-addvm' are:


--order(required) The order of the new VM instance.

--vmfile The VM file path of the new VM instance.

--tafconfig The TAF configure file path of the new VM instance.



--report_dir_vm The directory for the report on the virtual machine.

--report_dir_host The directory for the report on the host machine.

--snapshot The snapshot name.


The ‘n’ represents the order of the VM instance.





--nodefaults Specify this option to not use any default values. If this

option is not specified, default values will be used for

items that are not specified.

This example adds a new VM instance to the test instance at 14:00 by specifying only the –time and –

order option. It shows that a VM instance is added to the order number 1 with default values, then adds a

VM instance with order number 2 using a different TAF configure file.

[aya@osllx21 bin]$ ./configure_ete --addvm ETE.xml --time=14:00 --order=1

Adding/replacing <VM> to a test instance...



Time : 14:00



Run order: 1




System tests: all


CPP unit tests: all



---------------end-------------------

[aya@osllx21 bin]$ ./configure_ete --addvm ETE.xml --time=14:00 --order=2 --

tafconfig=/drive/ATST_TEST/Configs/TAF_config_Canary_8.xml

Adding/replacing <VM> to a test instance...



Time : 14:00



Run order: 1




System tests: all


CPP unit tests: all



Run order: 2





TAF config file: /drive/ATST_TEST/Configs/TAF_config_Canary_8.xml

System tests: all


CPP unit tests: all



---------------end-------------------

6.3.2.6 ‘--delvm’ command

Usage: ./configure_ete --deletevm < ETE configure file path > --time=<time> --orders=<run orders>

To remove a VM instance from a test instance, use the --deletevm parameter. The instance to remove

needs to be specified by test execution time and the order number of the VM instance.

[aya@osllx21 bin]$ ./configure_ete --deletevm ETE.xml --time=14:00 --order=2

VM at the run order 2 deleted.



Time : 14:00



Run order: 1




System tests: all


CPP unit tests: all



---------------end-------------------

6.3.2.7 ‘--list’ command

Usage: ./configure_ete --list < ETE configure file path >

To list all test instances use the --list parameter.

./configure_ete --list /home/qas/ATST_TEST/Configs/ete_config.xml

Listing ETE config file entries.

Time : 12:00

Final report directory: /home/qas/ATST_TEST/atst_test/FinalReport


Run order: 1

VM file: /drive/VMWareWorkstation/Centos66/Centos66.vmx

Snapshot name: Canary_8_Ready

TAF config file: /home/qas/ATST_TEST/Configs/taf_config.xml




Report directory on the host machine: /home/qas/ATST_TEST/atst_test/../Reports

Run order: 2

VM file: /drive/VMWareWorkstation/Centos66/Centos66_2.vmx

Snapshot name: Canary_8_Readyist




Run order: 3

VM file: /drive/VMWareWorkstation/Centos66/Centos66_3.vmx

Snapshot name: Canary_8_Ready




6.3.2.8 ‘--edit’ command

Usage: ./configure_ete --edit < ETE configure file path > <options>

To edit items in a test instance, use the –edit parameter.

The test instance must be specified by the execution time. Items that need to be updated should be

specified with the options listed below.

--emails Email addresses separated by ','. If '' is set then email

tags will be deleted.

--frd The path to the directory where final report will be

stored.

--vm(n) Virtual machine settings. Specify items <VM_FILE>,

<TAF_CONFIG>, <REPORT_DIR_PATH_ON_VM>,

<REPORT_DIR_PATH_ON_HOST>, <SNAPSHOT> separated by ','

without spaces. The number ‘n’ represents the order of the

VM instance.



TAF.


TAF.

In the example below, the emails of a test instance at 14:00 and System tests of the VM instance order 1

is updated.

[aya@osllx21 bin]$ ./configure_ete --edit ETE.xml --time=14:00 --systemtests_vm1=10- -

[email protected]

Editing the test instance (14:00)...



Time : 14:00



Run order: 1




System tests: 10-


CPP unit tests: all





---------------end-------------------

6.3.2.9 ‘--check’ command

Usage:./configure_ete –check < ETE configure file path >

Use the –check parameter to check an ETE configure file to see if all required items are set and can be

used by ETE. It checks whether the file is in the correct format and files and directories specified in the

ETE config file exist.

./configure_ete --check /home/qas/ATST_TEST/Configs/ete_config.xml

Checking ETE config file

Error:

instance (time:12:00) VM3 REPORT_DIR_PATH_ON_HOST tag is missing

Warning:

instance (time:12:00) VM1 VM_FILE (/drive/VMWareWorkstation/Centos66/Centos66.vmx)

doesn't exist.

instance (time:12:00) VM1 TAF_CONFIG (/home/qas/ATST_TEST/Configs/taf_config.xml)

doesn't exist.

instance (time:12:00) VM1 REPORT_DIR_PATH_ON_HOST

(/home/qas/ATST_TEST/atst_test/../Reports) doesn't exist.

6.3.2.10 ‘--copy’ command

Usage: ./configure_ete –copy <file path of the ETE configure file to copy> <new ETE configure file

path>

To copy and create a new ETE configure file, specify the –copy parameter followed by the path of the

ETE configure file to copy from and the new ETE configure file path.

[aya@osllx21 bin]$ ./configure_ete --copy ETE.xml ETE_2.xml


New ETE config file (ETE_2.xml) created.

6.3.2.11 ‘--help’ command

Usage: ./configure_ete --help

Specify –help to show the help message.

------------------------------ETE Config Editor------------------------------

This python program helps creating/editing ETE config file.

Usage: ./configure_ete <command> <options>


--new Create a new empty ETE config file.

--add Add a new test instance.

--copy Copy an ETE config file to create a new ETE config file.

--edit Update items.

--del Delete a test instance.

--addvm Add a new vm instance to a test instance.

--deletevm Delete vm instances from a test instance.

--list List current setting of a ETE config file



--check Check an ETE config file

--changetime Change the execution time of a test instance.



Specify –help with a command name to show the help message of that command.

./configure_ete –help <command name>

[aya@osllx21 bin]$ ./configure_ete --help add

Usage: ./configure_ete --add <file name> --time=<time> <options>

Create a new test instance in the specified ETE config file.

Available options are :


--frd The path to the directory where final report will be stored.

--emails Email addresses. If specified, final report will be sent.

Multiple address can be specified separated with ','.

--vm(n) Virtual machine settings. Specify items <VM_FILE>,<TAF_CONFIG>

<REPORT_DIR_PATH_ON_VM>,<REPORT_DIR_PATH_ON_HOST>,<SNAPSHOT>

separated by ',' without spaces.

Specify--vm with number of order. For example, if adding <VM>

with order number of 2, option name is --vm2.

--nodefaults Specify this option to not use any default values.

If this option is not specified, default values will be used

for items that are not specified.


Specify 'all' to run all tests.

--javaunittests_vm(n) Specify the IDs of Java unit tests which should be run by TAF.


--cppunittests_vm(n) Specify the IDs of CPP unit tests which should be run by TAF.


6.3.2.12 Interactive configure tool

The configure_tests program can be used to edit ETE configuration files interactively. Below is an

example of creating a new ETE configuration file then editing and checking it. Pressing Ctrl+D will

cancel the current editing process and go back to a previous menu. When editing, the changes that are

made to the ETE configuration will not be saved to the file until ‘Save’ is selected. ‘*’ will appear next to

the file name to indicate the configuration has unsaved changes. This program also makes use of the

default file $ATST_TEST/src/python/editor/ete_defaults.

[atst-qas@osllx25 bin]$ ./configure_tests

--------------configure_tests--------------

(Ctrl+D can be used to cancel current editing process)

Create New file or Edit existing file (New/Edit)? >n

Which config file to create? (T)AF or (E)TE)? >e

Creating new ETE config file.

Please specify the new ETE config file path >ete_config

note:Extension '.xml' is added to the specified file name

-------Editing ETE config file <ete_config.xml>-------

1:Add 2:Del 3:ChangeTime 4:Edit

5:List 6:Check 7:ListTaf 8:ChangeFile

9:Save 10:Copy 11:Quit 12:Help

>1

-----Adding new test instance-----

Time(UTC)? >12:00

VM1 VM file name? (default:/drive/VMWareMachines/ATST_READY/ATST_READY.vmx) >

VM1 TAF config file name? (default:/drive/ATST_TEST/Configs/TAF_config_HEAD.xml) >



VM1 Report folder on VM? (default:/mnt/hgfs/Reports) >

VM1 Report folder on host? (default:/drive/ATST_TEST/Reports) >

VM1 Snapshot name? (default:Ready Type 'None' to have no Snapshot set.) >

Specify SYSTEM_TESTS? (yes/No)>y

SYSTEM_TESTS for VM1?(default:all) >1-3

Specify JAVA_UNIT_TESTS? (yes/No)>y

JAVA_UNIT_TESTS for VM1?(default:all) >

Specify CPP_UNIT_TESTS? (yes/No)>y

CPP_UNIT_TESTS for VM1?(default:all) >a

Emails? (default:[email protected] Type 'None' to have no email addresses

set.) >

Final report directory? (default:/drive/ATST_TEST/FinalReport) >

New test instance added.

-------Editing ETE config file <ete_config.xml*>-------




>5

Time : 12:00



Run order: 1




System tests: 1-3

Java unit tests: all

CPP unit tests: a







>4

Which test instance to edit? (Time 12:00)

(Just hit enter to quit editing VM of the test instance) > 12:00

-----Editing a test instance (12:00)-----

1:Emails 2:Frd(Final report directory) 3:Vm(Edit a VM instance)

4:AddVm(Add VM) 5:DeleteVm(Delete a VM)

6:List(Show current setting of the teset instance)

Edit instance (12:00): What item to edit? >VM

------Editing VM1------

1:VMfile 2:Tafconfig 3:RepoVm 4:RepoHost

5:Snapshot 6:RunOrder 7:List 8:ListTaf

VM1: Which item to edit? >snapshot

Current Snapshot name: Ready

New snapshot name >Ready2

------Editing VM1------

1:VMfile 2:Tafconfig 3:RepoVm 4:RepoHost

5:Snapshot 6:RunOrder 7:List 8:ListTaf

VM1: Which item to edit? >

-----Editing a test instance (12:00)-----

1:Emails 2:Frd(Final report directory) 3:Vm(Edit a VM instance)

4:AddVm(Add VM) 5:DeleteVm(Delete a VM)

6:List(Show current setting of the teset instance)



Edit instance (12:00): What item to edit? >

Which test instance to edit? (Time 12:00)

(Just hit enter to quit editing VM of the test instance) >

Time not specified.





>5

Time : 12:00



Run order: 1


Snapshot name: Ready2


System tests: 1-3

Java unit tests: all

CPP unit tests: a







>6

ETE config file OK.





>listtaf

<<<< /drive/ATST_TEST/Configs/TAF_config_HEAD.xml START >>>>

ID:HEAD Checkout, Some tests.



Softwares:

CSF=HEAD

BASE=HEAD

QAS=HEAD

Unit tests:







System tests:



Install application:yes


baseDir = $ATST







USE_CPP = yes

















init-db:yes


<<<< /drive/ATST_TEST/Configs/TAF_config_HEAD.xml END >>>>





>save

ETE config file (ete_config.xml) is saved.

-------Editing ETE config file <ete_config.xml>-------




>quit

[atst-qas@osllx25 bin]$

6.4 TEST RESULTS AND REPORT GENERATION

Section 5.2.5 describes the report format that is produced for each test executed by the TAF. Once all

tests have been completed the reports are collated and then parsed to produce the final report. The final

report provides the details of virtual machine and TAF configure files used and the execution time. It

then shows a short summary of results for each of the types of test (unit and system). The summary

shows how many tests were executed, how many passed and how many failed. For each test failure

details of the failure are appended to the end of the report to aid in tracing the problem that occurred. An

example report is shown below.

Test Execution: Fri Jan 23 10:10:01 2015

Virtual machine: /drive/VMWareMachines/ATST_READY/ATST_READY.vmx


Results Summary:

Run Pass Fail

Unit 32 30 2

System 97 97 0


ID = REG_CANARY_8_ATCS_791.h

CVS ID = $Id: REG_CANARY_8_ATCS_791.h,v 1.1 2014/10/01 09:03:09 ChrisMayer Exp $

VM name = "/drive/VMWareMachines/ATST_READY/ATST_READY.vmx"

Snapshot = "Ready"



TAF ID = "HEAD

Type = unit

Time = 2015/01/23:11:59:14.361 GMT

Pass = False

Machine = qas01

Failures =

Locations =

REG_CANARY_8_ATCS_791.h[140]

Reasons =

Error: Expected (nanoStringOut[i]+gmt == d1->toNanoString()), found

("2014/09/15:12:34:56.1234567891 GMT" != "2014/09/15:12:34:56.123456789 GMT")

Versions =

[CSF=HEAD]

[Base=HEAD]



VM name = "/drive/VMWareMachines/ATST_READY/ATST_READY.vmx"

Snapshot = "Ready"

TAF ID = "HEAD

Type = unit

Time = 2015/01/23:11:59:19.472 GMT

Pass = False

Machine = qas01

Failures =

Locations =


Reasons =

Error: Expected (nanoStringOut[i] == d1->toNanoString()), found

("2014/09/15:12:34:56.1234567891 GMT" != "2014/09/15:12:34:56.123456789 GMT")

Versions =

[CSF=HEAD]

[Base=HEAD]

6.5 CONFIGURING A CRON JOB FOR ETE

You can set up a cron job to run the ETE every day at the time specified in an ETE configure file. The

example below shows a typical cron job setting.

*/30 * * * * export ATST_TEST=/drive/ATST_TEST/atst_test;filename=`eval date -u

+\%Y\%m\%d_\%H\%M\%S`"_ETE.log";/drive/ATST_TEST/atst_test/bin/ETE --

eteconfig=/drive/ATST_TEST/Configs/ETE_config.xml --vmuser=qas --vmpassword=password -

-emailpgm=mail > $ATST_TEST/etelog/$filename

With this cron job setting, the ETE will be called every 30 minutes. If it finds a test instance to execute

for the current time, then it will execute the test. In this cron job, the output will be stored in a file

$ATST_TEST/etelog/<file name>. The file name is constructed as YYYYmmdd_HHMMSS_ETE.log.



7. SETUP OF VMWARE VIRTUAL MACHINE

To ensure a repeatable set of tests are executed each time the testing procedure is carried out virtual

machines is used that provide the same starting point for each test run. A set of virtual machines should

be available, each with a specific setup that can be utilized by the ETE. The virtual machines should be

setup and snapshots taken so that after each test run the virtual machine is returned to a "clean" state, with

no properties or temporary files left over. Two copies of each virtual machine should be available to

allow for networked tests to take place (not network performance tests as these must be executed across

real networks).

7.1.1 Virtual Machine Snapshot Setup

Virtual machines should be created in pairs, one pair for each testing environment. A testing environment

is defined by the operating system running on it (and the versions of some packages), the postgres

database version running, the version of ICE, the version of Java and the version of the GCC compiler. If

any of these environmental setups are changed then another pair of virtual machines should be created for

testing purposes. By maintaining these sets of virtual machines it is easier to verify conditions that might

affect the failure of tests, and this method eliminates the possibility of tests being affected inadvertently

by upgrading operating system packages, or installing other software that appears to be unrelated but

actually isn't. The virtual machines and snapshots should follow a simple naming convention to easily

identify what setup is present on each instance.

7.1.2 Shared Folder

As the tests are executed the only information that should be retained are the test data files and reports. A

shared folder should be created in a virtual machine to store files and reports. Once the tests have all been

executed and the virtual machine has been shut down all of the test reports and data files are available to

the ETE for compilation of the results.

Each execution of the tests will result in a new sub-directory within the main shared folder so that no

previous sets of test reports and data files can be overwritten. Currently it is assumed that manual

archiving of the test data may be required over time but that is outside the scope of this document.

7.1.3 Remote Control of Virtual Machines

For the ETE to automate execution of tests it is necessary to be able to control the virtual machines

remotely. VMWare Workstation provides a remote control tool called ‘vmrun’. This provides the

following operations as command line tools, available for scripting

Power on virtual machines.

Power off virtual machines.

Copy files from host to guest.

Delete files from host to guest.

Rename files from host to guest.

Run programs and scripts in guest.

Using the ‘vmrun’ it is possible to run the required operations on a virtual machine from a host machine’s

terminal. This means the guest machine doesn't need to have any special software or scripts to execute

the tests, all execution is controlled externally.



8. UNIT TESTING

For the purposes of simplifying the DKIST testing procedure it has been decided that instead of

segregating unit tests and assembly tests they shall all be considered as unit tests. A description of the

definition of unit and assembly testing is presented below, after which all tests that fall under these

categories will be considered as unit tests.

Unit testing is a method by which individual units of source code, sets of one or more computer program

modules together with associated control data, usage procedures, and operating procedures, are tested to

determine if they are fit for use. Intuitively, one can view a unit as the smallest testable part of an

application, the level of a class in C++ and Java. Unit tests are usually created by programmers during

the development process and would be considered white box testing. Ideally, each test case is

independent from the others: substitutes like method stubs, mock objects, fakes and test harnesses can be

used to assist testing a module in isolation.

Assembly testing provides the next step up from unit testing. It demonstrates that modules (classes)

interact in a stable and proper manner as defined by the functional specifications. Assembly testing takes

as its input modules that have been unit tested, groups them in larger aggregates, applies the defined

assembly tests defined to those aggregates, and delivers as its output the integrated system ready for

system testing.

For the DKIST software unit testing we require separate testing suites for Java and C++. The selected test

frameworks that will be used for the unit tests are JUnit for the Java codebase and CxxTest for the C++

codebase. Both of these frameworks are open source and actively maintained, and CxxTest has

previously been used to perform some unit tests on the CSF C++ codebase. Neither of these frameworks

are intrusive to the code which allows the software tests to be developed alongside the current DKIST

CSF and Base stable versions without disruption.

Unit tests will only be carried out of CSF and Base code, there are no requirements to perform unit tests

on other systems or subsystems. However, due to the size of the CSF and Base and the effort constraints

it is currently not realistic to expect unit tests to be written for all classes. Therefore a selection of classes

that are considered critical have been selected for unit testing and tests will be written for both languages

(Java and C++).

8.1 WRITING A UNIT TEST

Wherever possible when writing a unit test the test must be added for both Java and C++. Only in the

case of testing differences in the two languages should separate tests be written. To write a new unit test

the following steps should be followed.

8.1.1 Writing a Java Unit Test

1) Identify the name of the new unit test. There is no restriction placed on the naming of tests but if the

name includes a short description of what the test covers it is easier to manage large sets of tests.

2) Create a new Java class under [TBD] the name of which is the same as decided in step 1.

3) Create a test method within the class file and annotate the test method with @org.junit.Test to notify

the JUnit test framework that this is a test method.



4) Write the test. JUnit provides many useful static methods in the Assert class to test for certain

conditions, which throw AssertionExceptions if the comparison fails. The table below lists some of the

Assert static methods available.

Statement Description

fail(String) Let the method fail. Might be used to check that a

certain part of the code is not reached. Or to have a

failing test before the test code is implemented.

assertTrue([message], boolean condition) Checks that the boolean condition is true.

assertsEquals([String message], expected,

actual)

Tests that two values are the same. Note: for arrays the

reference is checked not the content of the arrays.

assertsEquals([String message], expected,

actual, tolerance)

Test that float or double values match. The tolerance is

the number of decimals which must be the same.

assertNull([message], object) Checks that the object is null.

assertNotNull([message], object) Checks that the object is not null.

assertSame([String], expected, actual) Checks that both variables refer to the same object.

assertNotSame([String], expected, actual) Checks that both variables refer to different objects.

The ATSTUnitTestRunner (described in section 4.1) will form the correct report format from the test if it

fails.

5) Add the test to the makefile and then follow the instructions in section 8.4 to add the test to a test set.

8.1.2 Writing a C++ Unit Test

1) Identify the name of the new unit test. There is no restriction placed on the naming of tests but if the

name includes a short description of what the test covers it is easier to manage large sets of tests. If

testing the same functionality as that of a Java unit test then keep the name the same.

2) Create a new C++ header file under [TBD] the name of which is the same as decided in step 1.

3) C++ unit tests are not executed in the same way as JUnit tests, the C++ unit tests must be compiled into

executable files. First the header files are parsed and then the generated code is compiled. Open the

header file.

4) Create a test class which extends the class CxxTest::TestSuite. Add test methods within the class file.

5) Write the test. CxxTests provides several useful macros for assertions to test for conditions. The table

below lists the assertion macros provided by CxxTests.

Macro Description



TS_FAIL(message) Fail unconditionally.

TS_ASSERT(expr) Verify (expr) is true.

TS_ASSERT_EQUALS(x, y) Verify (x==y).

TS_ASSERT_SAME_DATA(x, y, size) Verify two buffers are equal.

TS_ASSERT_DELTA(x, y, d) Verify (x==y) up to d.

TS_ASSERT_DIFFERS(x, y) Verify !(x==y).

TS_ASSERT_LESS_THAN(x, y) Verify (x<y).

TS_ASSERT_LESS_THAN_EQUALS(x, y) Verify (x<=y).

TS_ASSERT_PREDICATE(R, x) Verify P(x).

TS_ASSERT_RELATION(R, x, y) Verify x R y.

TS_ASSERT_THROWS(expr, type) Verify that (expr) throws a specific type of exception.

TS_ASSERT_THROWS_EQUALS(expr,

arg, x, y)

Verify type and value of what (expr) throws.

TS_ASSERT_THROWS_ASSERT(expr,

arg, assertion)

Verify type and value of what (expr) throws.

TS_ASSERT_THROWS_ANYTHING(expr) Verify that (expr) throws an exception.

TS_ASSERT_THROWS_NOTHING(expr) Verify that (expr) doesn't throw anything.

TS_WARN(message) Print message as a warning.

TS_TRACE(message) Print message as an informational message.

6) Add the test to the makefile. When building the build system will produce an executable for the test

that can be run.

7) Follow the instructions in section 8.4 to add the test to a test set.

8.2 EXAMPLE JAVA UNIT TEST

The following example test is taken from the original CSF Java unit tests and is out of date and may not

work. This example should be replaced once we have written some updated test classes.

package atst.cs.test;

import java.util.*;

import org.junit.*;



import atst.cs.interfaces.*;

import atst.cs.services.*;

import atst.cs.data.*;

public class TestData {

@Test

public void attribute() {

Boolean[] bArr = { new Boolean(true), new Boolean(false), new

Boolean(true) };

Double[] dArr = { new Double(Double.MIN_VALUE), new Double(0.0d),

new Double(Double.MAX_VALUE) };

Integer[] iArr = { new Integer(Integer.MIN_VALUE), new

Integer(0),

new Integer(Integer.MAX_VALUE) };

String[] sArr = { "string 1", "string 2", "string 3" };

float[] f_Arr = { Float.MIN_VALUE, 0.0f, Float.MAX_VALUE };

int[] i_Arr = { Integer.MIN_VALUE, 0, Integer.MAX_VALUE };

long[] l_Arr = { Long.MIN_VALUE, 0L, Long.MAX_VALUE };

IAttribute a1, a2, a3, a4, a5, a6, a7, a8, a9, aA, aB, aC, aD;

a1 = new Attribute("valueless");

a2 = new Attribute("BoolArray", bArr);

a3 = new Attribute("DoubleArray", dArr);

a4 = new Attribute("IntArray", iArr);

a5 = new Attribute("string", "a string");

a6 = new Attribute("sArray", sArr);

a7 = new Attribute("bool", true);

a8 = new Attribute("double", Double.MAX_VALUE);

a9 = new Attribute("float", Float.MAX_VALUE);

aA = new Attribute("floatArray", f_Arr);

aB = new Attribute("intArray", i_Arr);

aC = new Attribute("long", Long.MAX_VALUE);

aD = new Attribute("longArray", l_Arr);

int i;

// NOTE: Name value returned is NOT qualified. Spec implied

otherwise

Assert.assertTrue("valueless".equals(a1.getName()));

Assert.assertTrue("BoolArray".equals(a2.getName()));

Assert.assertArrayEquals(bArr, a2.getBooleanArray());

Assert.assertTrue("DoubleArray".equals(a3.getName()));

Assert.assertArrayEquals(dArr, a3.getDoubleArray());

Assert.assertTrue("IntArray".equals(a4.getName()));

Assert.assertArrayEquals(iArr, a4.getIntegerArray());

Assert.assertTrue("string".equals(a5.getName()));

Assert.assertEquals("a string", a5.getStringValue());

Assert.assertTrue("sArray".equals(a6.getName()));

Assert.assertArrayEquals(sArr, a6.getStringArray());



// Attribute value is turned into TRUE/FALSE string

Assert.assertTrue("bool".equals(a7.getName()));

Assert.assertTrue(Boolean.parseBoolean(a7.getStringValue()));

Assert.assertTrue("double".equals(a8.getName()));

Assert.assertEquals(null, Double.MAX_VALUE,

a8.getDouble().doubleValue(),

0.0d);

Assert.assertTrue("float".equals(a9.getName()));

Assert

.assertEquals(null, Float.MAX_VALUE,

a9.getFloat().floatValue(), 0.0d);

// No way of extracting primitives, so test using corresponding

classes

Assert.assertTrue("floatArray".equals(aA.getName()));

Float[] vA = aA.getFloatArray();

Assert.assertTrue(vA.length == f_Arr.length);

i = 0;

for (Float f : vA) {

Assert.assertEquals(null, f.floatValue(), f_Arr[i++],

0.0d);

}

Assert.assertTrue("intArray".equals(aB.getName()));

Integer[] vB = aB.getIntegerArray();

Assert.assertTrue(vB.length == i_Arr.length);

i = 0;

for (Integer i2 : vB) {

Assert.assertEquals(i2.intValue(), i_Arr[i++]);

}

Assert.assertTrue("long".equals(aC.getName()));

Assert.assertEquals(Long.MAX_VALUE, aC.getLong().longValue());

Assert.assertTrue("longArray".equals(aD.getName()));

Long[] vD = aD.getLongArray();

Assert.assertTrue(vD.length == l_Arr.length);

i = 0;

for (Long l : vD) {

Assert.assertEquals(l.longValue(), l_Arr[i++]);

}

}

}

8.3 EXAMPLE C++ UNIT TEST

The following example test is taken from the original CSF C++ unit tests and is out of date and may not

work. This example should be replaced once we have written some updated test classes.

#ifndef UT001_DATATESTSUITE_H_

#define UT001_DATATESTSUITE_H_

#include "cxxtest/TestSuite.h"

#include "cs/Attribute.h"



#include "cs/IAttribute.h"

#include "cs/AttributeTable.h"

#include "cs/Configuration.h"

#include "cs/HealthReport.h"

#include "cs/HealthRecord.h"

#include "cs/Health.h"

#include <fstream>

#include <iostream>

using namespace atst::cs::data;

using namespace atst::cs::interfaces;

using namespace atst::cs::util;

using atst::cs::services::Health;

class DataTestSuite : public CxxTest::TestSuite {

public:

void testAttribute ( void ) {

std::cout << std::endl;

// Test the creation of attributes

pIAttribute a1, a2, a3, a4, a5, a6, a7, a8;

pIAttribute a10, a11, a12, a13, a14, a15;

shared_ptr<Attribute> a9;

TS_ASSERT_THROWS_NOTHING(a1 = Attribute::create());

TS_ASSERT_THROWS_NOTHING(a2 = Attribute::create("test1"));

TS_ASSERT_THROWS_NOTHING(a3 = Attribute::create("test2", true));

TS_ASSERT_THROWS_NOTHING(a4 = Attribute::create("test3", 1));

TS_ASSERT_THROWS_NOTHING(a5 = Attribute::create("test4", (int64_t)2L));

TS_ASSERT_THROWS_NOTHING(a6 = Attribute::create("test5", (double)0.1));

TS_ASSERT_THROWS_NOTHING(a7 = Attribute::create("test6", (float)10.5));

TS_ASSERT_THROWS_NOTHING(a8 = Attribute::create("test7", "value"));

std::vector<bool> v1;

v1.push_back(false);

v1.push_back(true);

TS_ASSERT_THROWS_NOTHING(a9

= shared_ptr<Attribute>(new Attribute("test8", v1)));

std::vector<int> v2;

v2.push_back(3);

v2.push_back(5);

TS_ASSERT_THROWS_NOTHING(a10 = Attribute::create("test9", v2));

std::vector<int64_t> v3;

v3.push_back((int64_t)10);

v3.push_back((int64_t)20);


std::vector<double> v4;

v4.push_back(5.3);

v4.push_back(7.1);


std::vector<float> v5;

v5.push_back(9.2);

v5.push_back(8.5);


std::vector<std::string> v6;

v6.push_back("value1");

v6.push_back("value2");




std::string s1 = "abc";

std::string s2 = "defg";

TS_ASSERT_THROWS_NOTHING(a15 = Attribute::create("test14", s1));

// Test the checking of attribute values

TS_ASSERT(a3->getBoolean() == true);

TS_ASSERT(a4->getInteger() == 1);

TS_ASSERT(a5->getLong() == (int64_t)2L);

TS_ASSERT(a6->getDouble() == 0.1);

TS_ASSERT(a7->getFloat() == 10.5);

TS_ASSERT(a8->getString() == "value");

TS_ASSERT(a9->getBooleanArray() == v1);

TS_ASSERT(a10->getIntegerArray() == v2);

TS_ASSERT(a11->getLongArray() == v3);

TS_ASSERT(a12->getDoubleArray() == v4);

TS_ASSERT(a13->getFloatArray() == v5);

TS_ASSERT(a14->getStringArray() == v6);

TS_ASSERT(a15->getString() == "abc");

// Test the name setting and getting

TS_ASSERT_THROWS_NOTHING(a1->setName("test99"));

TS_ASSERT(a1->getName() == "test99");

// Test lack of a value

TS_ASSERT_THROWS_NOTHING(a1->isEmpty());

TS_ASSERT(a1->isEmpty() == true);

// Test the value setting

TS_ASSERT_THROWS_NOTHING(a3->setValue(false));

TS_ASSERT_THROWS_NOTHING(a4->setValue(5));

TS_ASSERT_THROWS_NOTHING(a5->setValue((int64_t)10L));

TS_ASSERT_THROWS_NOTHING(a6->setValue(0.5));

TS_ASSERT_THROWS_NOTHING(a7->setValue(2.5));

TS_ASSERT_THROWS_NOTHING(a8->setValue("value2"));

TS_ASSERT_THROWS_NOTHING(a9->setValue(v6));






TS_ASSERT_THROWS_NOTHING(a15->setValue(s2));

// Recheck the attribute values

TS_ASSERT(a3->getBoolean() == false);

TS_ASSERT(a3->isEmpty() == false);

TS_ASSERT(a4->getInteger() == 5);

TS_ASSERT(a5->getLong() == (int64_t)10L);

TS_ASSERT(a6->getDouble() == 0.5);

TS_ASSERT(a7->getFloat() == 2.5);

TS_ASSERT(a8->getString() == "value2");


TS_ASSERT(a10->getFloatArray() == v5);

TS_ASSERT(a11->getDoubleArray() == v4);

TS_ASSERT(a12->getLongArray() == v3);

TS_ASSERT(a13->getIntegerArray() == v2);

TS_ASSERT(a14->getBooleanArray() == v1);

TS_ASSERT(a15->getString() == "defg");



// Check the toString and show methods complete

TS_ASSERT(a13->toString() == "test12: 3, 5");

TS_ASSERT_THROWS_NOTHING(a14->show("Testing attribute->show()"));

pIAttribute a16;

TS_ASSERT_THROWS_NOTHING(a16 = a9->clone());

TS_ASSERT(a16->getName() == "test8");


shared_ptr<Attribute> a_write;

TS_ASSERT_THROWS_NOTHING( a_write =

shared_ptr<Attribute>(new Attribute("name1",

"value1")));

std::ofstream outfile("writeData.out");

TS_ASSERT_THROWS_NOTHING( a_write->writeData( outfile ) );

std::ifstream infile("writeData.out");

std::tr1::shared_ptr<IAttribute> attr_in;

TS_ASSERT_THROWS_NOTHING( attr_in = Attribute::readData(infile));

TS_ASSERT(attr_in);

if (attr_in) {

TS_ASSERT( attr_in->getName() == "name1" );

TS_ASSERT( attr_in->getString() == "value1" );

}

pIAttribute attr_parse;

TS_ASSERT_THROWS_NOTHING( attr_parse =

Attribute::parseAttribute("name2\tvalue2") );

TS_ASSERT(attr_parse);

if (attr_parse) {

TS_ASSERT( attr_parse->getName() == "name2" );

TS_ASSERT( attr_parse->getString() == "value2" );

}

}

};

#endif /*UT001_DATATESTSUITE_H_*/

8.4 ADDING A UNIT TEST TO A TEST SET

To add the unit tests to a test set simply open the TAF configuration file for the test set that that this test

should be appended to (or multiple configuration files). Search for the "TESTS" section and then the

"UNIT" section. Add two new entries, one for the Java version and one for the C++ version.

<TESTS>

<UNIT>

<JAVA>test/java/unit_tests_001</JAVA>

<C++>test/c++/unit_tests_001</C++>

</UNIT>

</TESTS>

Individual unit test files can be added to as many TAF configuration files as required, there is no reason

not to run every unit test on every test run. Equally to keep certain test runs short it may not be necessary

to execute all unit tests.



9. SYSTEM TESTING

System testing is carried out on a fully integrated software system to verify that the system meets its

specified requirements. The system tests should have no knowledge of the inner design or logic of the

software that is being tested. The DKIST CSF provides a Jython interface to its public API which is ideal

for system tests as they can be written using the python language and kept away from the internal DKIST

codebase

9.1 WRITING A PYTHON SYSTEM TEST

A system test script is simply a Python file that subclasses the relevant testing classes described in section

4.3. This section provides a step by step method for writing a system test. This example will use the case

of posting and receiving an event. The full listing of this test can be found in section 9.2.

1) Identify the name of the new system test. There is no restriction placed on the naming of tests but if

the name includes a short description of what the test covers it is easier to manage large sets of tests. For

this example the test name is ATST_SYS_001_EventPost.

2) Create a new python file under [TBD] the name of which is the same as decided in step 1

(ATST_SYS_001_EventPost.py).

3) Some imports are required at the top of the file. These imports expose the API required for writing

system tests, all test scripts should start with these imports.

import sys

if sys.path.count(Misc.getRelease() + 'src/java/atst/tcs/test') == 0:

sys.path.append(Misc.getRelease() + 'src/java/atst/tcs/test')

from TestFrameworkFailure import TestFrameworkFailure

from TestFrameworkStep import TestFrameworkStep

from TestFramework import TestFramework

4) Additional imports for CSF specific functionality may be desired. The full CSF service set can be

imported.

from atst.cs.util import *

from atst.cs.services import *

from atst.cs.services.event import *

4) For each test step a new python class must be created that subclasses the TestFrameworkClass. It is a

good idea to use the name of the test and append the test step number to it, to ensure classes inside each

test script cannot inadvertently overwrite other scripts.

class ATST_SYS_001_EventPost_Step01( TestFrameworkClass ):

5) Implement the run method of the class. There is no restriction placed on what occurs inside the run

method but making use of the provided API defined in section 4.3 will make test scripts consistent and

easier to manage in the future.

def Run( self ):

6) In the case of our test we want to post an event and then listen for that event. To be able to do this we

need to post asynchronously using the PostAsync call with a delay of 1.0 seconds.



table = AttributeTable()

table.insert(Attribute("testItem", "true"))

self.PostAsync( "atst.testChannel", table, 1.0)

7) Now let's log a message to state the table has been posted and then wait to receive the event. The

Listen call will block until the event is received.

self.LogMessage("Test event has been posted")

response = self.Listen("atst.testChannel", [".testItem"])

8) Finally let's log the contents of the response and fail if the value is not true.

self.LogMessage("Received event: " + response.toString())

if (response.getString(".testItem") != "true"):

self.Fail("The testItem value was not true as expected")

9) Once the test steps have been completed it is necessary to register the test with the framework so that

the test executor can pick up and execute the script when necessary. First create a new test object.

ATST_SYS_001_EventPost = Test("ATST_SYS_001_EventPost")

10) Provide a description and add the test step.

ATST_SYS_001_EventPost.setDescription("Testing event posting and receiving.")

ATST_SYS_001_EventPost.addStep(ATST_SYS_001_EventPost_Step01)

11) Register the test with the framework

TestFramework.Register(ATST_SYS_001_EventPost)

12) The test script is now complete and ready for execution. The full script is presented in section 9.2

below.

9.2 EXAMPLE PYTHON SYSTEM TEST

The following example is a simple system test created for the purposes of this document. This example

should be updated with additional steps as the test framework software evolves.

import sys

if sys.path.count(Misc.getRelease() + 'src/java/atst/tcs/test') == 0:

sys.path.append(Misc.getRelease() + 'src/java/atst/tcs/test')

from TestFrameworkFailure import TestFrameworkFailure

from TestFrameworkStep import TestFrameworkStep

from TestFramework import TestFramework

from atst.cs.util import *

from atst.cs.services import *

from atst.cs.services.event import *

class ATST_SYS_001_EventPost_Step01( TestFrameworkClass ):

def Run( self ):

table = AttributeTable()

table.insert(Attribute("testItem", "true"))

self.PostAsync( "atst.testChannel", table, 1.0)

self.LogMessage("Test event has been posted")

response = self.Listen("atst.testChannel", [".testItem"])



self.LogMessage("Received event: " + response.toString())

if (response.getString(".testItem") != "true"):

self.Fail("The testItem value was not true as expected")

ATST_SYS_001_EventPost = Test("ATST_SYS_001_EventPost")

ATST_SYS_001_EventPost.setDescription("Testing event posting and receiving.")

ATST_SYS_001_EventPost.addStep(ATST_SYS_001_EventPost_Step01)

TestFramework.Register(ATST_SYS_001_EventPost)

9.3 ADDING A PYTHON SYSTEM TEST TO A TEST SET

To add system tests to a test set simply open the TAF configuration file for the test set that that this test

should be appended to (or multiple configuration files). Search for the "TESTS" section and then the

"SYSTEM" section. Add a new entry for the Python test.

<TESTS>

<SYSTEM>

<PYTHON>test/python/ATST_SYS_001_EventPost</PYTHON>

</SYSTEM>

</TESTS>

Individual system test files can be added to as many TAF configuration files as required, there is no

reason not to run every system test on every test run. Equally to keep certain test runs short it may not be

necessary to execute all system tests.



10. REGRESSION TESTING

Regression testing is used to determine any new bugs that have been introduced into a system after a new

release has been made. As part of the testing procedure any new bug that is found within DKIST

software should have a test written (unit, system or a combination) that firstly can be used to reproduce

the fault, and secondly can be used to verify the fault has been fixed. From the point at which the fault

has been verified as fixed the test becomes part of the regression test suite, and should continue to be

executed on every new release of the software to ensure the fault has not re-occurred. By definition every

DKIST test that is written to demonstrate a fault will become part of the regression test suite for future

releases.

By utilizing the Testing Automation Framework (section 5) and the E2E Test Executor (section 6)

regression testing should be a simple case of adding the test file to the list of defined tests for DKIST and

it will then be executed automatically when these frameworks are run.

10.1 SETTING UP A REGRESSION TEST RUN

To setup a regression test run the tester should first determine which tests are to be run. This may range

from a small set of tests looking for a specific problem up to every test that has been written for the

specific system that is undergoing regression tests. If we use the example of CSF then it is quite likely

that the list of tests will include all available unit tests and CSF system tests.

Once the tests have been decided upon a TAF configuration file is created to setup the test run (see

section 5.3). For the case of CSF the configuration file would define the version of CSF to checkout only,

no other software modules should be included. The test scripts are then added along with the desired

location for test reports. An example configuration file is presented below:

<TAF>

<ID>Regression 01</ID>

<REPORT>/mount/reports/regression</REPORT>

<SOFTWARE>

<CSF>Canary_5</CSF>

</SOFTWARE>

<TESTS>

<UNIT>

<JAVA>UnitTest1</JAVA>

<JAVA>UnitTest2</JAVA>

<C++>UnitTest1</C++>

<C++>UnitTest2</C++>

</UNIT>

<SYSTEM>

<PYTHON>SystemTest1</PYTHON>

<PYTHON>SystemTest2</PYTHON>

</SYSTEM>

</TESTS>

</TAF>

Once the TAF configuration file has been created the tester can run the regression tests immediately by

simply executing the TAF and passing the configuration file to it. If it is preferable to automate the

execution of the regression tests, perhaps to schedule the execution for the same time each day then the

file can be added to the ETE (see section 6.3).



11. ETE TEST SERVER

An ETE environment has been set up on an ETE server schwabe. Currently there is one VMWare

workstation virtual machine under the home directory of username ‘qas’. It has been set up to checkout

the trunk version of csf, base and qas modules and run system tests and unit tests every day.

Path of the VMware workstation client: /home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx

There is a snapshot ETEReady. The snapshot was taken when the virtual machine was set up so that it is

ready to install atst csf and base modules of version Canary_8.

The atst_test module is checked out to /home/qas/ATST_TEST/atst_test.

The directory for the TAF report directory, where the report for each test script will be kept, is

/home/qas/ATST_TEST/TAFreports. This directory is shared with the virtual client. On the virtual client

machine, the path for this directory is /mnt/hgfs/Reports. Figure shows the setting of the shared folder of

the virtual machine.

Figure 2 Shared folder setting

The cron job for the ETE is set up as below.



*/30 * * * * export ATST_TEST=/home/qas/ATST_TEST/atst_test;filename=`eval date -u

+\%Y\%m\%d_\%H\%M\%S`"_ETE.log";/home/qas/ATST_TEST/atst_test/bin/ETE --

eteconfig=/home/qas/ATST_TEST/configs/ETE_config_centos66.xml --vmuser=atst-qas --

vmpassword=<password> --emailpgm=mail > /home/qas/ATST_TEST/ETElogs/$filename

The ETE is called every 30 minutes. The ETE configure file is

/home/qas/ATST_TEST/configs/ETE_config_centos66.xml and it is set as shown below. The test will be

executed every day at 09:00 UTC and the result email will be sent.

<ETE>


<EXECUTION>

<VM>

<VM_FILE>/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx</VM_FILE>

<TAF_CONFIG>/home/qas/ATST_TEST/configs/TAF_config_HEAD_Install_and_test.xml</TAF_CONF

IG>


<REPORT_DIR_PATH_ON_HOST>/home/qas/ATST_TEST/TAFreports</REPORT_DIR_PATH_ON_HOS

T>

<SNAPSHOT>ETEready</SNAPSHOT>

<SYSTEM_TESTS>1-51,56,58,73-77,79-94</SYSTEM_TESTS>

<JAVA_UNIT_TESTS>10-15</JAVA_UNIT_TESTS>

<CPP_UNIT_TESTS>7,10-16</CPP_UNIT_TESTS>


</VM>

</EXECUTION>

<EMAIL_ADDRESS>[email protected]</EMAIL_ADDRESS>

<FINAL_REPORT_DIR>/home/qas/ATST_TEST/ETEFinalReports</FINAL_REPORT_DIR>

</instance>

</ETE>

The TAF configure file is also stored under /home/qas/ATST_TEST/configs directory.

Below is an example of email sent from the ETE. The email subject will include [FAIL] if there are any

failed tests.

Subject: Test report for ETE_config_centos66.xml [FAIL]

Test Execution: Fri Feb 20 09:00:01 2015

Virtual machine: /home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx

TAF config file:

/home/qas/ATST_TEST/configs/TAF_config_HEAD_Install_and_test.xml

Results Summary:

Run Pass Fail

Unit 14 12 2

System 74 73 1


ID = SYS_REQ_4.1_0101

CVS ID = $Id: SYS_REQ_4.1_0101.py,v 1.5 2014/09/10 09:20:43 ChrisMayer Exp $

VM name = /home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx

Snapshot = ETEready

TAF ID = Installing HEAD and test

Type = system

Time = 2015-02-20 09:09:25.260

Pass = False

Machine = qas01



Failures =

Failure 1

Reason = The container SYSREQ410101JAVA2 is not registered.

Step = 7

Location = StepCheckContainerIsRegistered

Versions =

[CSF=HEAD]

[Base=HEAD]



VM name = "/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx"

Snapshot = "ETEready"

TAF ID = "Installing

Type = unit

Time = 2015/02/20:10:37:24.247 GMT

Pass = False

Machine = qas01

Failures =

Locations =


Reasons =

Error: Expected (nanoStringOut[i]+gmt == d1->toNanoString()), found

("2014/09/15:12:34:56.1234567891 GMT" != "2014/09/15:12:34:56.123456789 GMT")

Versions =

[CSF=HEAD]

[Base=HEAD]



VM name = "/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx"

Snapshot = "ETEready"

TAF ID = "Installing

Type = unit

Time = 2015/02/20:10:37:29.381 GMT

Pass = False

Machine = qas01

Failures =

Locations =


Reasons =

Error: Expected (nanoStringOut[i] == d1->toNanoString()), found

("2014/09/15:12:34:56.1234567891 GMT" != "2014/09/15:12:34:56.123456789 GMT")

Versions =

[CSF=HEAD]

[Base=HEAD]

--------------End of report--------------

To run another test, for example testing another versions of csf and base, just add a new test instance to

the ETE configure file. One thing to be careful of is the execution time. Because the cron job is run every

30 minutes, the test execution time should be set to 0 or 30 minutes past the hour. Another thing to note is

that the ETE can’t run tests if the virtual machine is running. For example, the test at 9:00 UTC takes

about one and half hours to complete. The new test’s time should be set to 11:00 UTC or later if the same

virtual machine is used.

To edit configure files on schwabe, log in to the server as a user within the qas group. The new TAF

configure file can be created using configure_taf or configure_tests tools (see 5.4.1). After logging in to

schwabe, make sure to set the environment variable ATST_TEST to /home/qas/ATST_TEST/atst_test.



The default file taf_defaults for the TAF configure file tools has been set up for the virtual machine

QAS01_CentOS66.vmx. Part of the file is shown below.

INST_DIR=/opt

REPORT=/mnt/hgfs/Reports

CREATE_DEVEL_ACTION=--make-all

MAKE_ACTION=build_all

POSTGRESQL=postgresql-9.3

INITDB=Yes

ICESERVICE=Yes

INSTALL_APP=Yes

#Software defaults should be placed between 'SOFTWARES_START' and

'SOFTWARES_END'

#It should be specified as <Software name>=<Version>

SOFTWARES_START

CSF=HEAD

BASE=HEAD

QAS=HEAD

SOFTWARES_END

#Site config default items should be placed between 'SITE_CONFIG_START' and

'SITE_CONFIG_END'

SITE_CONFIG_START

baseDir=$ATST

ATST_RELEASE_FLAG=trunk

ATST_RUN_ENVIRON=/opt/atst/var

ATST_JAVA_HOME=/opt/java8

ATST_JLIB_DIRS=none

USE_CPP=yes

ATST_CLIB_DIRS=/usr/pgsql-9.3/lib/:/usr/local/lib/

ATST_ARCHIVE_DB_HOST=localhost

ATST_ID_DB_HOST=localhost

ATST_LOG_DB_HOST=localhost

ATST_PROPERTY_DB_HOST=localhost

ATST_PARAM_SET_DB_HOST=localhost

ATST_SCRIPT_DB_HOST=localhost

ATST_HEADER_DB_HOST=localhost

ATST_ICE_VERSION=3.5.1

ATST_ICE_CONNECTION_HOST=localhost

ATST_ICE_EVENTS_HOST=localhost

ATST_ALARM_DB_HOST=localhost

ATST_BULK_DATA_DB_HOST=localhost

SITE_CONFIG_END

…

The ETE configure file on schwabe can be edited using configure_ete or configure_tests tools (see 6.3.2).

The default file ete_defaults for the ETE configure file tools has been set up for the ETE server.

[aya@schwabe bin]$ more ../src/python/editor/ete_defaults

VM_FILE=/home/qas/vmware/QAS01_CentOS66/QAS01_CentOS66.vmx

#TAF_CONFIG=


REPORT_DIR_PATH_ON_HOST=/home/qas/ATST_TEST/TAFreports

FINAL_REPORT_DIR=/home/qas/ATST_TEST/ETEFinalReports



#EMAIL_ADDRESS=

#SNAPSHOT=

#SYSTEM_TESTS=

#JAVA_UNIT_TESTS=

#CPP_UNIT_TESTS=



12. FUTURE UPDATES

Below is a list of possible future updates.

1. The TAF should be able to install atst using pkgDevel.

2. The TAF should be able to do ‘cvs update’. Currently it can only do ‘cvs checkout’.

3. The TAF should be able to checkout any modules specified in a TAF configuration file. Currently

it can only checkout CSF, BASE, TCS and QAS.

4. The TAF should be updated to be able to take a snapshot of a virtual machine in case of test

failure.

5. TestFrameworkStep.py should implement a method to submit a configuration with timeout.



13. REFERENCES

[1] Wampler, W. & Goodrich, B., ATST Software Design Document, SPEC-0014, Rev. 0.3

[2] Greer, A. & Mayer, C., ATST TCS Acceptance Test Plan, SPEC-xxxx, Rev. A

[3] Kneale, R., ATST Glossary and Acronym List, SPEC-0012, Rev. D1

[4] Wampler, S. & Goodrich, B., ATST Common Services Software Design Document Vol. I User

Manual, SPEC-0022-01, Rev. A4

[5] Greer, A., Hubbard, J. & Wampler, S., Java Engineering Screens User Manual, TN-0089, Rev.

3.0/A11/A

quality assurance software test plan · quality assurance software test plan alan greer, chris...

Documents