guest lecture from motorola

Motorola Computer Group 1

Software Testing: Software Testing: Gets the Bugs out, and MoreGets the Bugs out, and More

Dean Cai

Motorola Computer Group


Software Testing in the Life Cycle Software Testing in the Life Cycle of a Productof a Product

Life cycle of a telecommunication platform product

Software testing in different stages of a product life cycle


Challenges of Developing a Telco-Challenges of Developing a Telco-grade, High Availability Platformgrade, High Availability Platform

Five-9’s and six-9’s

High Availability

Fault Tolerant

Open architecture issues

Linux source level compatibility

User support in an open source environment

Out-of-box experience

auto detect and auto configure in an user defined environment


The Software Design GoalsThe Software Design Goals

Unified implementation running on all platforms and multiple operating systems.

Scaleable software architecture for the continuous advancement of system availability.

Ease of integration of telecom applications.

Ease of integration of telecom specific subsystems to expand the platform offering.


The Software SystemThe Software System

An Architectural view of the software system in a complex telecommunication platform

Example: Motorola Computer Group’s CPX system


A Sample System ApplicationA Sample System Application

NetworkManager

Packet NetworkATM

IP

CPXNetworkElements


CPX System HA FeaturesCPX System HA Features

Compact PCIBus Segment A

Compact PCIBus Segment B

I/O Slots

CPU-A CPU-B

Bridge-B Bridge-A

I/O Slots

• Hot Swappable CPU and I/O boards Dual CompactPCI backplanes• Hot Swappable drives, fans, power supplies, bridges and alarm panel• Chassis status & control via HSC• Domain control via HSC• Telco alarms• 2N, N+1 and N+m configurations


Advanced High Availability SWAdvanced High Availability SWCustomer

ApplicationPrograms

CustomerDevelopedScripts &Utilities

EventManagement

APIsMessaging &

CheckpointingAlarm

Management

Distributed Messaging

Inter-Host Comm.& Checkpointing

Heartbeat &SwitchoverProtocols

SystemConfiguration

& EventManager

SystemAlarm

ManagerEvent Management

Checkpointing Agent

Policies,Rules,

Actions,Methods

HAUtility

Programs

Web Server/ Graphical

User &Interface

PCI ServicesHSC Driver

ENUM Handler

Event Mgr. Driver HAAwareDrivers

BackplaneMessaging

Driver

MIB

SNMPMIBMIB

I/O Bus Driver

Cluster Agent


Application LayerApplication LayerCustomerApplicationPrograms

CustomerDevelopedScripts &Utilities

Web Server/ Graphical

User &InterfaceMIB

SNMPMIBMIB

• SNMP v3 Agent

• Monolithic or Master-Subagent models

• Standard MIB2 Support

•

UCD extensions for some sysadmin functions

• Motorola MIB for Event, Domain, Alarm and Chassis Management

• Graphical User Interface to Event, Domain, Alarm and Chassis Management

• Customer provided applications, utilities and scripts


Application Interface LayerApplication Interface LayerEvent

Management

APIsMessaging &

CheckpointingAlarm

Management

Policies,Rules,

Actions,Methods

HAUtility

Programs

• APIs (libraries) for event and Alarm Management

• API to general messaging Application interface

• Simplified checkpointing interface

•

Database of system component/object definitions

• Policies and Rules for managing system state transitions

• Actions and methods/programs for handling state transitions

• Utility programs for Event, Domain, Alarm and Chassis Management


Event Manager LayerEvent Manager LayerDistributed Messaging

SystemConfiguration

& EventManager

SystemAlarm

ManagerEvent Management

Checkpointing Agent

Cluster Agent

• Clustering agent synchronizes Databases, Rules and actions to multiple systems.

• Checkpointing agent provides synchronization services for applications and Event Manager.

• System Configuration & Event Manager provides configurable system event and availability management.

• System Alarm manager controls and maintains chassis alarms, LEDs, Fans and power supplies.


Operating System LayerOperating System LayerInter-Host Comm.& Checkpointing

Heartbeat &SwitchoverProtocols

PCI ServicesHSC Driver

ENUM Handler

Event Mgr. Driver HAAwareDrivers

BackplaneMessaging

DriverI/O Bus Driver

• PCI services for dynamic insertion/ deletion of devices.

• HSC driver for chassis status and control.

• ENUM handler for fielding slot events.

• HA Aware drivers specification and examples to include fault detection,

fault diagnosis, fault recovery and active - standby configuration.

• Backplane messaging driver and utilities to provide low bandwidth board to board communication utilizing the PCI bus.

• Event Manager driver funnels event reporting to management layer.

• Inter-Host communication for synchronization of state and checkpoint data between redundant drivers and stacks.


Strategies of Testing a Complex Strategies of Testing a Complex Software System (1)Software System (1)

Unit test

Integration test Mix of vendor FW, in-house FW, vendor

middleware, in-house middleware, utility commands, and open source OS

Example: pxe driver and hsc driver confliction



static intpxe_open(struct device *dev){// __asm__ __volatile__ (".align 2\n\t" ".byte 241");

if (net_debug)printk("Entered pxe_open()\n");

if (request_irq(dev->irq, &pxe_interrupt, SA_SHIRQ, "Universal Network Driver", dev)) {

/*

if (request_irq(dev->irq, &pxe_interrupt, 0, "Universal Network Driver", dev)) {

*/return -EAGAIN;

}



System test System level functionalities System stress environment User friendly evaluation


Test Planning (1)Test Planning (1)

The process of generating a test plan MRD SRD Design specification Test requirement Test plan generation Review and approval



An example of a test plan



Test scenario design Scenario for function coverage Scenario for system stress Scenario for fault injection, fail-over, and

fail recovery



Test tool developing API test Utilities and demo applications using

user accessible APIs Hooks in application and device driver

for fault injection Remote test tools, such as SNMP


Test ExecutionTest Execution

DVT (hardware, firmware)

Software installation and configuration

Utility command based testing (feature verification, code validation)

Stress test

Expose potential timing issues (HW and SW)

Expose potential system resource allocation problems

Memory leak

User application simulation (user scenario test, API validation)


Problem Reporting, Resolving, and Problem Reporting, Resolving, and VerifyingVerifying

Quality control from software testing perspective

The process flow of our in-house problem reporting system (CPR)


Test Management and AutomationTest Management and Automation

Ways to improve the quality of testing– Process control– Automation

An sample approach:

System Test Management Framework


Project GoalsProject Goals

Increase test coverage Increase test efficiency Reduce System Test Cycle Time Reduce preparation time and learning

curve Increase test traceability Built-in process control


Description of the Project (1)Description of the Project (1)

Identify requirements for test coverage Analyze current test coverage Examine current test tools Identify the needs for improving existing

tools Identify the needs for new tools



Improve the existing tools Develop new tools Select a test framework package (TETware, HP-

VUE, LabView, etc.) Consolidate tools into an user-friendly framework Develop an user front-end of the framework for

test configuration and deployment



Build a test database for collecting and archiving test data

develop a test plan generator develop a test report generator Develop a CM package for tool

development and maintenance Staff training


Analyze Current Test CoverageAnalyze Current Test Coverage

Overall system coverage DVT coverage (CPU, backplane, IO slots,

alarm panels, power supply, etc.) Software functionality coverage Software integration coverage System functionality coverage System performance coverage


HCT

AIXSuiteports

GoAheadTest Suite

Florian

100% coverage

HA testsHoles

Overlaps

No-needsManual

test

Overall System CoverageOverall System Coverage


OSAppl DriverAPI

Unit Test

Bd Level DVT

OS Integ Test

System TestConfig. 1

Config. 2

Config. X

Software Functionality CoverageSoftware Functionality Coverage


Test Framework ArchitectureTest Framework Architecture

User Login

HW Configurator

SW Configurator

Test Monitor

ToolDownload/Rm mount

DataCollector

ReportGenerator

TestEvent Log

HW ConfigDatabase

SW ConfigDatabase

Test DocDatabase

Test DataDatabase

Go

Test ToolLib

Test ToolStudio

Test PlanGenerator

User Interface


Platform CPX8221

CPX1205

Gemini 2.0

PlatformConfigurator

Suite#1

Suite#2

Suite#X

Test SuiteConfigurator

1. Login on Test Server2. Select Platform3. Enter platform config data4. Select test suite(s) to run5. Enter test suite config data

Sample Test Flow (1)Sample Test Flow (1)


Targethostname

TH-1

TH-2

TH-X

6. Enter target host name7. Test suite download from Server to Target 8. Target console opens on the server login9. Test suite starts on Target10. Test data transfer from Target to Server

Server

Target

Testsuite

Go

Server

Target

Testdata

Sample Test Flow (2)Sample Test Flow (2)


Test Framework SelectionTest Framework Selection

TETware - Unix Script type, no GUI LabView HP-VUE VisualBASIC SNMP Manager

guest lecture from motorola

Documents