1
March 3, 2011
Automating SCA Testing with Spectra CX
INTRODUCTION
Spectra CX is a model-driven development tool that forms the backbone of the Software Communications Architecture (SCA) development process.
Spectra CX (SCX) reduces development risk, accelerates development and improves the software quality of SCA applications and platforms through SCA validation and automatic generation of SCA artefacts.
The SCA Test feature facilitates a key element in this development process by providing industrial strength SCA component test code generation, execution of tests on host or target systems, and detailed reporting of test results.
2
AGENDA
Overview of the SCA Test feature
Benefits of SCA Test
How it works
How you use SCA TestWorkflow
Demonstration
Customization and Extensibility
3
Validation versus Testing
XML Descriptors
Binaries
Validation(Static)
Test(Dynamic)
ValidationReport
TestCases
ValidationRules
TestReport
4
SCA Model
Spectra CX – SCA Test
Model-based testing framework for the SCAPlug-in feature for Spectra CX 3.2.X
Provides radio developers with a standalone capability for unit testing SCA components
On-host or on-target using TargetLoader Implementation independent, black box, testing using JUnits and CORBA
Includes test cases for the Component Lifecycle interfaceSupports user-defined test cases
5
Tests – Base Application Interfaces
CF::Port Test connectPort succeeds for a valid connectionTest connectPort fails with an InvalidPort exception (error code 1) when called with an invalid object referenceTest connectPort fails with an InvalidPort exception (error code 1) when called with an object reference which can't be narrowed to the correct typeTest connectPort succeeds where number of connections <= maximum (where maximum is finite)Test connectPort fails with an OccupiedPort exception when the number of connections exceeds the maximum (where maximum is finite)Test disconnectPort succeeds for a valid connectionTest disconnectPort fails with an InvalidPort exception (error code 2) when the connection ID is invalidTest disconnectPort fails with an InvalidPort exception (error code 2) when attempting to disconnect an already disconnected port
CF::LifeCycle Test initialize succeedsNote: No negative tests as these are implementation dependant
Test releaseObject succeeds
CF::TestableObject Test runTest fails with an UnknownTest exception when trying to run an unknown testTest runTest fails with an UnknownProperties exception when invalid properties are passed to runTestTest runTest succeeds when valid properties are passed to a valid test
CF::PortSupplier Test getPort succeeds for a valid portTest getPort fails with an UnknownPort exception for an invalid port
CF::PropertySet Test configure of invalid simple propertyTest query of invalid simple propertyTest configure of valid simple propertyTest configure of valid simple property with invalid typesTest query of valid simple propertyTest query of exec paramTest configure then query of valid simple propertyTest query returns all properties if no property specifiedTest query returns only specified properties (1 property)Test query of external allocation properties.Test configure of properties with range (< min, min, max, > max)
CF::Resource Test identifier matches value expectedTest start completes successfullyTest stop completes successfullyTest configure works when resource is stoppedTest query works when resource is stoppedTest start works when resource is stopped
CF::ResourceFactory Test identifier matches value expectedTest createResource succeeds for a valid resourceTest createResource returns a reference to the same resource when a resource is created twiceTest releaseResource succeeds for a valid resourceTest releaseResource fails for an invalid resourceTest releaseResource only releases resources with multiple references when the reference count is 1Test shutdown succeeds
Note CF::Port tests are currently generated by the same script which generates tests for CF::PortSupplier
Note: Similar tests could also be written for SimpleSequence, Struct and StructSequence properties
6
Tests – Base Device Interfaces
CF::Device Test usageStage attribute is initially set to IDLETest adminState attribute is initially set to UNLOCKEDTest softwareProfile attribute contains the correct valueTest label attribute contains the correct valueTest allocateCapacity with an invalid property IDTest deallocateCapacity with an invalid property IDTest allocateCapacity with invalid typesTest deallocateCapacity with invalidTypesTest releaseObject succeeds (including checking relevant state changes are published to a mock IDM, sub devices released, device manager de-registration)
CF::LoadableDevice Test load succeeds for a valid mock executableTest load fails when called with invalid LoadType for device (multiple tests)Test multiple load of an executable succeedsTest load executable fails when called with an invalid filenameTest unload succeeds when given a valid filenameTest unload succeeds after unloading multiple loaded executables
Test unload with duplicate valid file names (should fail the 2nd time)
CF::ExecutableDevice Test execute succeeds for a valid mock executableTest execute succeeds for a valid executable with valid propertiesTest execute succeeds for a valid executable with valid optionsTest execute fails when invoked with an invalid nameTest execute fails when invoked with invalid propertiesTest execute fails when invoked with an invalid optionsTest execute fails when invoked with an executable which hasn't been loaded firstTest terminate succeeds with valid PIDTest terminate fails with invalid PIDTest terminate fails with PID of process which has terminatedTest double terminate of PID
CF::AggregateDevice Test addDevice succeeds for a valid deviceTest removeDevice succeeds for a valid deviceTest behaviour of addDevice when adding a duplicate deviceTest removeDevice fails for a device which has not been addedTest addDevice for a a nil device failsTest removeDevice for a nil device failsTest addDevice failure results in a FAILURE_ALARM log record being generated if appropriateTest removeDevice failure results in a FAILURE_ALARM log record being generated if appropriateTest child devices are released when releaseObject is called (will require a mock device)
7
Benefits of Unit Testing
By automating the process of testing SCA components and applications, SCA Test significantly accelerates the testing cycle, improves user productivity and establishes high quality software from the very beginning of the development cycle
SCA Test can be used to test the compliance of software that may have been hand written, generated automatically using a Spectra CX Code Generator, or generated using a 3rd party tool.
Testing early on and throughout the development cycle reduces the risks that affect time to market, cost and software quality.
By exercising components at the beginning of the development cycle in a runtime environment, and by continuing to test as components are refined, developers can be assured continuous SCA compliance up to development cycle completion.
8
Benefits of SCA Test
SDR/SCA-specific model-based testingProvides a test capability that is optimized for SDR/SCA
Testing based on the current design and implementation
Completely integrated with the Spectra CX development environment, All SCA Test compile, build and execute actions can be initiated from the Spectra CX
environment.
Users can prepare the runtime environment for execution and observe their code running on host and/or target.
Developers benefit from the simplicity of using only one tool for modeling, validation, code generation, execution, and test.
Tests can be executed on host or targetUnit testing can even begin with only a basic CORBA connection to the component
allowing testing to begin before a full platform with a live SCA operating environment is available.
Multiple implementations of a component can be tested to ensure proper operation on all defined implementation platforms.
9
How it Works 10
SCA TestFeature
Eclipse JUnitFramework (Java)
SCA TestTarget Loader (C++)
ComponentUnder Test
Communicates withvia CORBA
Instantiates
GeneratesJUnits
Instantiates& Terminates
Component
Host
Host or Target
TargetLoader
In an SCA system typically the operating environment and more specifically the platform elements take care of instantiating components and or devices. Since it is desirable to test components before this may be available or without the overhead of the SCA operating environment we provide TargetLoader software.
TargetLoader is responsible for loading the component under test into the appropriate environment.
Typically there will be one TargetLoader for each target environment
A TargetLoader can be referenced by multiple component implementations -- don’t have to model the TargetLoader for each component implementation Once a TargetLoader is running, it can be reused for multiple test executions
11
SCA Test Workflow
Generate
Compile
Execute
Evaluate
12
PrepareFor each
component to be tested
Create test project(s)/model(s)
Build the TargetLoader(s)
Run the TargetLoader(s)
Model the TargetLoader(s)
Generating Component Tests
SCA Test generates a JUnit test for a component’s LifeCycle, PortSupplier, PropertySet, ResourceFactory, Resource and Device interfaces
Any component implementation can be designated for automatic testing.
A comprehensive suite of tests will be automatically generated for each component.
The generated test harness includes all the code needed for connecting to and exercising, through CORBA, the resource interface on the component under test -- an SCA Core Framework is not required.
13
Compiling Component Tests
The target may be an actual SDR platform or the host development environment, depending on the compiler and CORBA tooling that is available for the test harness.
The generated testing software can be fully integrated into the project’s build process.
Spectra CX can be directed to automatically build the generated test harness into an executable, ready to run on host or target.
14
Executing Component Tests
SCA Test generates the JUnits which interact with the Target Loader
Target Loader deploys and instantiates the component implementation and test harness binaries on the host or target platform
Users simply initiate the test from Spectra CXOnce initiated, execution of the tests defined for SCA resource compliance is carried out by the JUnit through the CORBA environment, without need for an SCA Core Framework to be present
Results of the tests are recorded by the JUnit
15
Evaluating Test Results
Upon completion of the tests, a unified report is presented in the SCX user interface
This report can then be used to correct any issues identified and/or prove compliance with the SCA standard for components that support the LifeCycle, PortSupplier, PropertySet, ResourceFactory, Resource or Device interfaces
A standard JUnit report (xml) is also generatedThis can be converted into HTML and other formats
16
DEMONSTRATION
17
Customization and Extensibility
Spectra CX uses open standards to translate model information to test code and gives users full access to the code template.
Users can modify testsEngineering teams can optimize the template to accommodate their environment if they need to.
Provide customer extensible framework for unit testing a component
Generate empty tests for all provides ports methodsCustomers can add their own JUnit tests
Generate Target Loaders from the tool using build configurations
Provides support for the customer’s target platform(s)
18
Adding Unit Tests for an SCA Component
1. Create a new Java Project
2. Create a new class for the test cases
3. Configure the build path for the project
4. Add a JUnit library
5. Add a default constructor to the test class
6. Add test cases
19
Create a new Java Project
Select ‘File > New >Java Project’ while in the Java Perspective.
In the ‘New Java Project’ wizard set the ‘Project Name’. Leave the other values as their defaults. Click the ‘Next’ button.
20
Set the project parameters
On the ‘Java Settings’ page switch to the ‘Project’ page.
Click the ‘Add…’ button and add the project that contains your generated test code.
21
Create a new class for the test cases
Right-click on the “src” folder in the new Java project and select ‘New > Class’.
22
Set the class fields
In the ‘New Java Class’ wizard set the ‘Package’, ‘Name’ and ‘Superclass’ fields.
The ‘Superclass’ field should be set to the implementation test class from the generated test code.
23
Configure the build path for the project
Configure the build path for the project so that the class will not have compilation errors.
Select the new Java project and right-click, then select ‘Build Path > Configure Build Path…’
24
Add a JUnit Library
Switch to the ‘Libraries’ page and click on ‘Add Library…’.
Select the ‘JUnit’ library
Make sure that the ‘JUnit Library Version’ is set to JUnit 3.
25
Add a default constructor to your class
For example, the following shows a default constructor that makes use of the default behavior and changes the path to the component under test binary. If the component is generated by Spectra CX then your path will resemble this pattern “../<generated src project name>/” + hostFileName.
public EncryptorUnitTests() throws Exception {super();hostFileName = "../EncryptDecrypt_SRC/" + hostFileName;
}
Test out your extension using Run As > JUnit Test
26
SUMMARY
Spectra CX 3.2 includes the SCA Test feature which provides for:Standalone, unit testing of componentsImplementation independent, black box, testing using JUnitsOn-host or on-target using TargetLoader in place of a Core FrameworkBasic component lifecycle interface testing -- 32 test casesThe addition of user-defined test cases (JUnits)
Future releases will expand the test coverage with cases forComplete Lifecycle interface testing – 129+ test casesComponents, Assemblies, and ApplicationsPlatforms: Core Framework, Services, …
We will also improve the testing capability withSCA-specific JUnit framework – easier to create test casesSupport testing profiles -- standard (e.g. SCA, SCA-Next) and customSupport for test hardware
27
Additional Information 28
For additional information on PrismTech’s Spectra products and services:
E-mail:
Website:
http://www.prismtech.com