testing strategy white paper

AcceleratedSAP document.doc - January 21, 2000

Testing Strategy White PaperOverview

This document establishes an overall strategy for testing that will be followed throughout the life of the project. It can be used as a copy to be taken and adapted to the special needs of a project.

Purpose of a Testing StrategyThis testing strategy defines the requirements and goals of the R/3 configuration, determines the tools and methods used to check that the system responds correctly, determines how and when the test will be performed and recommends how the approval process should occur.

When developing your testing strategy you have to consider the test scope and the test constraints from your project plan and the implementation scope.

Testing should be considered and planned very early in an implementation project to setup additional tools (automated test tools), train the project team, and still have testing time to fit into the overall schedule.

ProcedureAcceleratedSAP uses an iterative testing approach that builds upon itself. As you begin to configure the R/3 System, you will be unit testing your configuration settings to ensure that the system responds correctly. As you get into Final configuration, you will be testing the business processes to ensure that each process responds correctly. Additionally, there is an independent test performed at each stage to ensure integrity.

In parallel you are testing your own development (interfaces, conversions, enhancements) to ensure the correctness of the programs. After you have finished the configuration and development you will perform the Final Integration Test to ensure that the entire system including external components works correctly.

To avoid double work start testing the system with small test scenarios during Baseline and Final Configuration. These short testing scenarios will be augmented to get confirmation scenarios at the end of each configuration cycle. These bigger test scenarios can be used as a basis to develop integrated test scenarios for the Final Integration Test.

Planning for TestingDifferent levels of testing require different levels of planning. Unit testing is at the transaction level and the testing is simple with perhaps different sets of data. At the other end of the spectrum, integration testing can be very complex. The planning and testing time for integration testing increases as the number of processes, interfaces, enhancements and configuration changes increase. If the implementation project is large and complex then having extra resources becomes necessary. A successful integration test is dependent on a thorough plan.

Types of TestingThere are different types of testing that occur during the project, and each plays a critical role in the project’s ultimate success. The testing strategy includes:

» Unit testing

» Scenario testing

» Development testing

» Integration testing

» System testing

Copyright © 2000 SAP AG. All rights reserved of 10


» Technical tests

» Performance tests

» User acceptance testing

Unit TestingDefinition

This is the lowest level of testing where the program or transaction is tested and evaluated for errors. Unit testing is normally the first test that is completed during the configuration effort, and is focused towards the program’s inner functions, rather than towards the integration.

Testing focus:

» Master data

» Business processes

Examples: You can test master data by creating a Customer or a customer hierarchy. Creating a standard sales order is an example of a basic business transaction in the Sales and Distribution module.

Besides application testing unit testing is also performed during development and even during technical system testing (please refer to Development Testing and System Testing).

Phase and Work PackageUnit testing will be performed as part of the baseline and final configuration work packages of the realization phase. You find several tasks for defining test cases and developing test plans during Baseline Configuration and Final Scope Configuration.

Test OrganizationDuring configuration, unit testing is done whenever it is needed. If there is a special need to organize and plan unit testing because of the project requirements, update the BPML worksheets with planning information and resource allocation.

Create the Baseline Scope worksheet for baseline configuration and testing. In the Responsibilities/Tester column you define the responsible person for testing. The Testing Dates/Plan-Actual columns are used to plan and keep track of the testing days. The status of each single unit test can be updated in the Status column. Use the Comments column to add short documentation about the test.

For Final Scope configuration and testing create Cycle Scope 1 to n worksheets, and maintain the Responsibilities/Tester and the Testing Dates/Plan-Actual columns to plan testing.

Test DocumentationBecause unit testing concentrates on testing single transactions during the ongoing configuration process there is normally no need to develop detailed test documentation. For simple transactions testing can be done straightforward during configuration and the results (status T = Tested and any comments) can be recorded in the corresponding BPML worksheet.

However, some transactions are very complex involving multiple screens, functions and variations to run. Those transactions (i.e. Sales order) can be documented and tested with a BPP, maintaining the test section with test conditions and variations of the standard transaction.

Roles



Consultants configure the system during Baseline configuration and perform unit testing for Baseline. During Final Scope configuration customer team members learn transactions and configuration settings, and perform some unit testing.

SystemUnit testing is performed in the development system, using a different testing client. Therefore the configuration settings have to be copied to the other client before testing.

Make sure that the systems you select for testing are consistent with your overall system and client landscape.

Scenario TestingDefinition

During the configuration there is a need to test chains of transactions that flow together and which reflect important business processes and scenarios.

Testing focus:

» Multiple transactions within one enterprise area

» Workflow

» Business processes which cross enterprise areas

Examples: (of small scenarios)

» Standard sales order to a delivery to invoicing

» Invoicing to account document and posting

During subsequent integration testing these small scenarios can be used to build larger end-to-end scenarios.

Phase and Work PackageScenario testing will be performed as part of the Baseline and Final Scope Configuration work packages of the realization phase. You find several tasks for defining test cases and developing test plans during Baseline Configuration and Final Scope Configuration.

We also recommend using test scenarios for Baseline confirmation and for Final Scope confirmation. According to ASAP at the end of Baseline Configuration as well as at the end of Final Scope configuration the most important business processes should be tested, presented and confirmed using Confirmation Scenarios. Therefore existing test scenarios can be linked and adapted to create end-to-end-scenarios that reflect the overall business flow.

Test OrganizationTo organize and follow up the scenario testing during baseline and final configuration use the BPML. Enter additional lines for each process or scenario, which has to be tested. You can either maintain the Master List or the Baseline/Cycle Scope worksheets. Consider that you will lose all existing update information, if you create a new version of Baseline or Cycle Scope sheets from the updated Master List.

Update the testing dates and the responsibilities in the Baseline and Cycle Scope worksheets. The status of the scenario test can be updated in the Status column on process group or scenario level (T = Tested).

Use the Status column also to document that you confirmed the scenarios used for showing the integration in the system for your configured processes (A = Approved).



Test DocumentationTo document scenario testing it is recommended using the Test Scenario template entering every single step (transaction) with input and output data. These documents can also be used for confirmation of your integrated scenarios at the end of the Baseline Configuration and for the confirmation of each cycle during Final Scope Configuration.

TIP:

In the Roadmap tasks for the definition of test cases or in the accelerator list you find a list with predefined test scenarios. These test scenarios are delivered to support you in creating your own test scenarios. You can choose every scenario that meets your business, copy it and adapt it according to your needs. Create as many variations as you need to test your several business cases. There is no automated link to the BPML, but you can link the test scenario manually into the column for test scenarios (in Baseline Scope, Cycle Scope, and Integration Test).

RolesThe business process owner is involved in creating baseline scenarios along with the application consultant. Normally, the application consultant conducts the presentation to get the baseline confirmed. For the subsequent scenario testing and confirmation for each cycle during Final Configuration the business process owner should take on more responsibilities (i.e. defining the test scenarios).

SystemFor scenario testing and especially for running the confirmation scenarios it is recommended to use a special test client in the development system.

Development Testing Definition

Unit testing of your customer specific development.

» Conversions

» Interfaces – Third-party

» Interfaces – Legacy

» Interfaces – Online (EDI, ALE, Internet)

» Enhancements (User-exits and other code enhancements)

» Reports

» SAPScript forms

After development and unit testing is completed all customer specific programs and forms have to be included in the Final Integration Test.

Phase and Work PackageTesting of development is done in the realization phase during the development of conversions, interfaces, enhancements, reports, and forms (see corresponding work packages).

Test Organization



To organize the testing of all developments use the Development Master List and update responsibilities and dates for testing. Create links to all corresponding documents (Functional Specification, Test Scenario).

Test Documentation To define test requirements, test data and test results during the unit test, use the test section of the Functional Specification document. For integration test each development has to be included in a special test scenario to ensure that the whole business flow works (see Final Integration Test).

RolesThe developer along with the application consultant and/or the business process owner mainly does unit testing of development.

SystemFor unit testing of your development use a special testing client in the development system, which provides enough sample data for testing.

Integration TestingDefinition

Final integration testing is accomplished through the execution of predefined business flows, or scenarios, that emulate how the system will run your business. These business flows, using migrated data from the pre-existing systems, will be performed in a multifaceted computing environment comprised of R/3, third-party software, system interfaces and various hardware and software components. It is this environment that builds the necessary level of confidence that the solution is complete and will perform in your business.

Final integration tests need to be an evolutionary process that is driven from your previous testing efforts. The test cases and scenarios that were used for Baseline and Final Configuration need to be reviewed and enhanced for the integrated test. These selected cases can be combined to represent a business process flow such as a revenue cycle or a material acquisition cycle. Problems encountered during these efforts also need to be tested under an integrated environment. Final Integration test cannot be viewed as an independent, but as a capstone effort that brings together and builds upon all previous testing efforts.

Integration testing should be allotted approximately twenty-five percent (25%) of the total time used during the Realization Phase for configuration and testing.

Final Integration testing focuses on cross-functional integration points, as well as end-to-end business processes. A well defined Final Integration test plan starts with the testing of the cross-functional integration points (touch points) and ends with the end-to-end testing of critical business processes identified within the Business Blueprint.

Integration test should also include testing the organizational flow, i.e. how the several organizational units are involved in one process, how they communicate, how the information and document transfer is handled.

Integration Test 1 and 2Final Integration testing is recommended to be done in two iterations. The first iteration (Integration Test 1) concentrates on testing all important business processes inside the R/3 system, starting with touch point scenarios and ending with end-to-end-scenarios.



If customer specific development like user-exits and transactions is already finalized and unit tested, it should be included in the first iteration. Authorizations and user roles should be also tested in the Integration Test 1.

Integration Test 2, as a second iteration, focuses on the most important cross-enterprise scenarios with touch points to external components, including testing of conversions, interfaces, reports, forms (SAPScript), EDI, ALE, and the necessary authorizations.

Phase and Work PackageAlthough there is some integration testing done during the confirmation sessions for Baseline and for Final Scope Confirmation, most of the integration testing takes place at the end of the realization phase in the Final Integration Test.

Test OrganizationIntegration testing, as stated in the Overview, is the most complex testing in a R/3 implementation project. It is mainly because the R/3 application is an integrated system. The integration test planning involves the following.

Recommend Test Leader for test organizationNominate a Test Leader. For smaller projects the Project Manager can fulfill this role. However, for a larger and more complex project a separate person who has already been a participant on the project would be able to focus on the Integration testing activities. It is the Test Leader who organizes the test team, and runs the workshops that accomplish the following tasks.

Establish test systemA test system, also called the Quality Assurance system (QA), should be setup by the technical team. It is necessary to have this system up and running before any integration test activity can be performed. The QA system should have been technically tested and the plan to transport configuration from the Development system should be completed.

Recommend test usersBuild an integration test team to ensure integration thinking and development of integrated test scenarios. Nominate test users. In many cases the team members can fulfill this role. However, if the project has been lengthy or if the company is large it is a good idea to involve extended team members from the user community. This involvement by extended team members serves two purposes: it trains them and helps to get their buy-in. It is highly recommended that authentic user authorizations and logins be used for the integration testing.

Recommend one room for integration testingOrganize a separate room for the integration test. Setup personal computers, networking, printers and any other necessary hardware in this room. Establish a schedule of the necessary test users for each part of the integration test. For example, if you are testing the invoicing to cash part of a scenario then make certain that the appropriate SD and FI team members are scheduled for the room.

If space and hardware is limited then have a detailed time schedule for each part of the test. The Test Leader should coordinate and facilitate throughout the different module teams to make sure that the integration test is smooth.

Recommend cut-off criteriaIt is easy to fall into a continuous loop of integration testing. However, a cut-off date and criteria should be established at the start of integration testing. The most critical testing scenarios should be used.

Scheduling and time estimates



Many of these preliminary tasks for integration testing should be completed in a timely manner. It is important to have the proper testers and systems available. During integration testing configuration changes may be required – enough time for these tasks should also be scheduled.

Preparing system and test dataThe critical element in an integrated test is common data. It is almost impossible to demonstrate integration without using the same data. Thus, a chart-of-accounts, business partners, materials, etc. must all be defined and properly configured as a part of the integrated test. Similarly, documents and control information (company codes, plants, organizational hierarchy, etc.) must function properly during the integrated test. A good deal of planning is required to conduct these tests (i.e. similar to planning to go into production). Successful completion of integrated tests is one of the precursors for going into production.

Test DocumentationUse the Integration Test Scenario template to define your test cases.

Use the BPML Integration Test worksheets (Integration Test 1 and Integration Test 2) to create the Integration Test plan. The following steps are a manual process:

» Insert lines with names of the test scenarios being tested (need only scenario or process group level)

» Maintain priorities for each scenario and select only high priority test scenarios, if you cannot test everything. Test priority should be based on:

» Frequency (How often does the process occur?)

» Impact (What is the effect if the process is down?)

» Difficulty (What is the probability that problems occur?)

Concentrate on test cases which are difficult (use the 80/20 rule: 80% of test cases have few problems, and 20% have most of the problems).

» Update testing dates

» Assign the appropriate test user(s) for each test scenario

» Complete the Status column. (Test, Re-test and Approved)

TIP:

In the Roadmap tasks for the definition of test cases or in the accelerator list you find a list with predefined test scenarios. There are also several real-life integrated test scenarios, which show how an integration test could be performed. These test scenarios are delivered to support you in creating your own test scenarios. You can choose every scenario that meets your business, copy it and adapt it according to your needs. Create as many variations as you need to test your several business cases. There is no automated link to the BPML, but you can link the test scenario manually into the column for test scenarios (Integration Test 1 and 2).

RolesAs the knowledge transfer from the R/3 business process experts to the project team approaches an apex, responsibility for the integration test becomes that of the business process users and power users. This also instills a sense of ownership with the business process users and should make it easier to get acceptance at the end of the phase.

There is a change in the make up of the testing teams between the configuration cycles and integration testing. As the test becomes more integrated, the test team should include team members from every enterprise area being tested. This will ensure that all processes are properly tested. Involve extended teams members to ensure you meet the company’s business requirements.



SystemQuality Assurance System is needed for Integration testing. Master data from legacy systems should be converted onto the Q/A system. All configurations should be frozen on the Development system and transported onto the QA system.

System Testing Definition

System testing consists of technical tests and performance related tests. The technical tests aim to validate that the technical components of the production environment are working properly. These tests include validating the system administration procedures, failure recovery, and disaster recovery. The performance related tests include volume and stress testing of business transactions and business transaction input / output using printer and fax devices.

Technical Test» Failure Test

» Disaster Recovery Test

» Backup and Restore Test

» System Administration Test

» Printing and Fax Test

» Going Live Check

Performance Test» Volume Test

» Stress Test

» Batch cycle test

» Month/quarter/year end processing

Technical tests are rather basis oriented, whereas performance tests cover basis and application aspects. Performance tests should concentrate on stress test, i.e. stressing the system with all components involved in certain scenarios until it performs to predefined values (throughput). Volume testing is needed for some processes or transactions, if the throughput of one process/transaction has to meet predefined requirements (i.e. creating large amounts of sales orders, daily interfaces, etc.)

Phase and Work PackageTechnical and performance tests are planned and prepared in the Realization phase under System Management. They are conducted in the Final Preparation phase under System Management.

Test Organization and DocumentationThere is no special tool provided by ASAP to organize technical testing. You can use the project plan to plan your several types of technical system test. Within the Roadmap steps you will find detailed information on how to test.

Use the Stress Test Strategy paper attached to the corresponding Roadmap task to help in performing the stress test. There is also an additional accelerator for planning, performing and documenting stress and volume testing attached to the Roadmap tasks.



RolesTechnical Team Lead (along with application team members).

System Testing can be prepared in the Quality Assurance system, but must run on the Production system.

User Acceptance TestingDefinition

Normally there is no special need for a User Acceptance Test, because the system should have been accepted by the confirmation of the Final Integration Testing. This is the reason why it is not included in the ASAP Roadmap. If you need to perform a separate User Acceptance Test, you can choose some of the Integration Test Scenarios.

Phase and Work packageFinal Preparation (optional, no Roadmap work package)

Test Organization and DocumentationUse some of your existing Integration Test Scenarios to perform and document the test.

RolesPower User, End Users.

SystemQuality Assurance system and Production system

Overview on how and when to perform testing

Baseline Configuration Final Configuration Integration Test

IT 1 IT 2

Development of Conversions, Interfaces, ...

Unit Test: BPP with test conditions

Scenario Test: Test Scenario template

Unit Test Development: Functional specification test section



Integrated test scenario

IT scenario with develop.

Type of Testing

Work Package or Activity Tools Roles System

Unit testing Baseline Configuration

Final Configuration

BPP with test conditions

Consultant Development system: QA environment

Scenario testing

Baseline Configuration

Final Scope Configuration

Test Scenario template

Business Process Owner

Development system: QA environment

Development testing

Develop Conversion Programs / Develop Interface Programs / Develop Enhancements / Create Reports / Create Forms

Test section of Functional Specification

Developer


Development system: QA environment

Integration testing

Baseline Confirmation

Final Scope Confirmation

Final Integration Test

Test Scenario template


Quality assurance system

Technical testing

System Management Checklists Technical Consultant / T. Team Lead

Productive system

Performance testing

System Management Stress/Volume test template

Technical Team Lead / Business Process Owner

QA system

Productive system


testing strategy white paper

Documents