microsoft powerpoint - mitpm_khoreddy_testing_print_final

Managing Independent Software Managing Independent Software Testing Project Testing Project -- A Case Study A Case Study from Multimedia Super Corridorfrom Multimedia Super Corridor

Khor Cheng San (SADC Manager)Khor Cheng San (SADC Manager)Azdilizan Abdullah (Senior Executive)Azdilizan Abdullah (Senior Executive)Solution Assessment and Development Centre (SADC),Solution Assessment and Development Centre (SADC),MSC Innovation CentreMSC Innovation Centre2 August, 20052 August, 2005

MSC Innovation Centre

Copyright Protected 2005 MDC

Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor



Table of ContentsObjectives of This Case StudyIntroduction to Solution Assessment and Development Centre (SADC)Test Project Management (TPM) MethodologyTest Project Management (TPM) ProcessesQuality Management System (QMS) FrameworkTest Methodology Framework (TMF)Case Study: Test Project OverviewCase Study: Test Project PhasesCase Study: Lessons Learned




Objectives of This Case Study

Present our experience in using PMBOK® and CMMI combination with our own testing processes that is the SADC Test Methodology Framework (SADC TMF) in managing independent software testing services to the stakeholders.

The SADC TMF encapsulates the various components required for test execution and test management, and is the basis for conducting Test Projects in the SADC.




SADC offers independent application verification & validation services, at a fraction (“cost-effective”) of the cost of building and maintaining an internal software test/QA department. Our Added-value proposition are:

Objective analysisFills the gap in staff resource Assures the delivery of reliable & robust application with proven SADC test methodologies and processes (QA/QC functions) “Do what they do best”

Solution Assessment & Development Centre (SADC)

“The solution for product perfection”




Solution Assessment & Development Centre (SADC)

“The solution for product perfection”




Test Project Management (TPM) Methodology

The SADC’s Project Management methodology applies recognized project management methods and techniques that are built upon

Project Management Institute's Project Management Body of Knowledge (PMBOK®) concepts. Software Engineering Institute’s Capability Maturity Model®Integration (CMMI) model.

This PMBOK® and CMMI combination culminates in the SADC’s Quality Management System (QMS) Framework and Test Methodology Framework (TMF) is to provide the basis for the SADC’s Test project teams to carry out their work in a well-structured manner to produce high quality output.




Test Project Management (TPM) Processes

The concept of our Project Management methodology is illustrated below:

INITIATIONPLANNING

CONTROL

EXECUTION

CLOSURE

Test Project Management Processes

The Test Project Management Processes of Initiation, Planning, Execution, Control and Closure are the type of processes occurring throughout a test project, keeping in balance the competing demands for scope, time, cost and quality.




Test Project Initiation

Initiation is the process of formally authorizing a new test project or that an existing test project should continue into its next phase.


Test Project Planning

Planning is an ongoing effort throughout the life of the test project and is considered the most important phase in test project management.

Test Project Execution

Test Project execution will be guided by the test plan developed in the Test Research/Analysis/Design Phase.





Test Project Control

Test Project control will involve the regular review of metrics and report status in order to identify variances from the test planned project baseline so that corrective action can be taken when necessary.

Test Project Closure

Test Project closure is the last major phase of a test project’s life cycle. It is performed once all stated test project objectives have been met and the client has formally accepted the test project’s deliverables.




Quality Management System (QMS) Framework

The QMS contains a set of processes, guidelines, templates, checklists and forms that will be tailored for the purpose of the Test Project.




Test Methodology Framework (TMF)

SADC Test Methodology Framework (SADC TMF)

SADC SADC TMFTMF

TestProcesses

Procedures & Templates

Test Methodologies

Standards& Guidelines

CM DatabaseOf Test Projects

SADC-CMDefect Management

SADC Test Methodology Framework (SADC TMF)

SADC SADC TMFTMF

TestProcessesTestProcesses

Procedures & TemplatesProcedures & Templates

Test MethodologiesTest Methodologies

Standards& GuidelinesStandards& Guidelines

CM DatabaseOf Test ProjectsCM DatabaseOf Test Projects

SADC-CMSADC-CMDefect Management

Defect Management




Documentation Components of SADC TMF

SADC Test Management Framework: Documentation Artifacts

SADC Test SADC Test MethodologiesMethodologies

SADCTest Methodologies

SADCConfigurationManagement

SADC Standards andGuidelines for

Test and Integration

Template Documents forSADC Test Projects

CM Database ofTest Projects

SADC Test Management Framework: Documentation Artifacts



SADCTest Methodologies

SADCConfigurationManagement

SADC Standards andGuidelines for

Test and Integration

Template Documents forSADC Test Projects

CM Database ofTest Projects

SADC Test Methodologies contains details of the Test Methodologies to be employed in the test execution and test management of TestProjects in the SADC.




Industry / Market – Airline

The Challenge – Client A wanted to performance test their Web Based System with realistic workloads to determine whether the system can handle the projected user load of 600 concurrent connection (10% of total user load) with a total estimated user base of 7200 crew (1200 Technical Crew and 6000 Cabin Crew) .

The Solution - SADC uses Rational Suite TestStudio™ to performance test Client A Web Based System with incremental concurrent user load of 1, 50, 100, 250, 500, 600 and 750 (to stress the system) virtual users. The performance test results will enable the client to accurately predict how their critical (widely used) functions will perform under projected user load.

Case Study: Test Project Overview




Testing HighlightsPerformance testing ensures proper scalability and identifies other performance problems in a real-life business environment.Performance testing uncovers a significant performance slowdown, which could be resolved by re-configuring and re-tuning of the identified root causes during the performance test.

The BenefitsImproved system quality and performance by pinpointing performance bottlenecks and other potential problems during performance tests.Highlighted specific area of code in the application where changes might significantly improve performance.





SADC Test Methodology

The overall test methodology employed by SADC for conducting the performance test is based on the Concurrent Users Load and Automated Simulation paradigm. This method is also referred to as the Automated Performance Test.

WorkloadModel

Automated testscripts f or applications ofthe SUT to beperf ormance-tested.These scripts areplay ed back during test execution,and assigned to VUs.

Applicationsyst em

components

Work-day Period

Resourcemonitoragents

Performance Test MethodologyNew Paradigm in Load Test/Stress Test: Virtual Users & Automated Simulation

Virtual Users (VUs)System processes w hichtakes the placeof actual users.

PerformanceTestServer

Performance Test CAST Tool

Generates required number of VUs, each VU executing the desired apps-functions according to the defined W orkload M odel.

Selected serversare monitored for its resource utilization. Resource UtilizationInformation ( % RAM and %CPU )relayed back to the Performance Test Server.

SUT underPerformance Test

Actual apps transactions

Response Time & Server ResourceUtilization data arecaptured, generated,and analyzed.

ApplicationUsage

Distribution

M odels full-load productionenv ironment of the SUTto be performance-tested.This is transcribed fromthe Application UsageDistribution.Workload M odel used asbasis to generate VUsand execute requiredautomated test scripts.

RelevantTest Plans,Test Procedures.

WorkloadModel

Automated testscripts f or applications ofthe SUT to beperf ormance-tested.These scripts areplay ed back during test execution,and assigned to VUs.

Applicationsyst em

components

Work-day Period

Applicationsyst em

components

Work-day Period



Performance Test MethodologyNew Paradigm in Load Test/Stress Test: Virtual Users & Automated Simulation

Virtual Users (VUs)System processes w hichtakes the placeof actual users.

PerformanceTestServer

Performance Test CAST Tool

Generates required number of VUs, each VU executing the desired apps-functions according to the defined W orkload M odel.

Selected serversare monitored for its resource utilization. Resource UtilizationInformation ( % RAM and %CPU )relayed back to the Performance Test Server.

SUT underPerformance Test

Actual apps transactions

Response Time & Server ResourceUtilization data arecaptured, generated,and analyzed.

ApplicationUsage

Distribution

M odels full-load productionenv ironment of the SUTto be performance-tested.This is transcribed fromthe Application UsageDistribution.Workload M odel used asbasis to generate VUsand execute requiredautomated test scripts.

RelevantTest Plans,Test Procedures.





How SADC Managed the Test Project





Pre-Project Test

Requirement

Research/ Analysis

Test Design

Implement Design and Test Setup

Test Execution and

Test Management

Test Assessment

Test Project Closure

Test Project Phases

•Define test project requirements.•Define Testing Scope•Define Test Project Timeline Estimation•Define Test Criteria•Agree on Communication Channels•Agree on Reporting Procedures, Method and FrequencyK

ey A

ctiv

ity

Method/Tools

•Analysis of test project requirements & relevant Client A documentation.•Analysis & mobilization of test execution requirements –manpower, test tools (hardware, software).•Design test configuration, test procedures, test scenarios, test cases.•Develop Workload Distribution Model (Load-Test Model).•Develop Test Plan/Test Procedures.

•Set up of SADC Test Equipments at test site •Set up of SADC Test Tools •Test Data Preparation •Automated Performance Test Scripts Generation •Pre-Test Execution to check all the automated test scripts are functioning as per requirement.

•Execute Test Procedures/Test Scenarios.•Manage Test Execution.•Record results of test execution.•Monitor & Capture various resource utilization data.

•Analyze Test Results•Final Test Assessment Report Documentation.•Final Test Assessment Report Briefing.

•Test Environment clean-up.•Test Project close-off (archive test artifacts; documentation, test datastores)

Case Study: Test Project Phases

Output

Input

Test Project Charter

Test Project Proposal

Test Plan Test Assessment Report

Certificate of Acceptance

Test PlanTest Project Charter

Test Assessment Report

Business Requirement

Test Suites Scripts and Test-related Documents




Case Study: Lessons Learned

The combination of PMBOK® and CMMI in SADC’s Quality Management System (QMS) Framework and Test Methodology Framework (TMF), provides a well-structured manner in producing high quality output and managing test projects.

Challenges CommunicationEnforcementTrade-off




For further information, please visit us at www.mscinnovationcentre.com.my/sadc

Khor Cheng San ([email protected])

&Azdilizan Abdullah

([email protected])




FOR YOUR ATTENTION

microsoft powerpoint - mitpm_khoreddy_testing_print_final

Documents

test management

test project planningplanning

existing test project

test project phases

test execution

project managementmethodology

test planned project

test projects deliverables