e-book...proper end-to-end testing essentially implies that we are talking about a bpml (business...
TRANSCRIPT
©KTern
1
E-Book – What you should know about SAP S/4HANA Testing Automation
Table of Contents Executive Summary ..................................................................................................................................... 2
What is SAP S/4HANA Testing? ................................................................................................................... 2
What is the SAP S/4HANA Test Lifecycle? ................................................................................................... 2
Problems in SAP S/4HANA Testing .............................................................................................................. 5
A Devil in Disguise – Testing Failures in SAP S/4HANA projects .................................................................. 6
5 Steps for S/4HANA Testing Automation ................................................................................................... 7
Nice To Haves along with Testing Automation ........................................................................................... 9
KTern Features for S/4HANA Testing Automation .................................................................................... 10
8.1 Test Case Identification ..................................................................................................................... 10
8.2 Test Case Execution ........................................................................................................................... 10
8.3 Test Case Reporting ........................................................................................................................... 14
8.4 VVV Scoring (Value, Volume and Velocity Analysis) .......................................................................... 14
8.5 Sign Off Wizard .................................................................................................................................. 14
Next Steps .................................................................................................................................................. 15
About KTern ............................................................................................................................................... 15
©KTern
2
E-Book – What you should know about SAP S/4HANA Testing Automation
Executive Summary
SAP S/4HANA is an intelligent ERP powered by SAP HANA and SAP Leonardo Platform. SAP S/4HANA provides solutions with tendrils and impacts every nook and corner of its enterprise records and processes. Hence, testing is mandatory and remains critical for every single SAP ECC to SAP S/4HANA migration project.
Enterprises must ensure high quality end-to-end business processes even though they are often committed to continuous change. This results in problems surrounding timely releases and business process quality. In addition, test managers face tremendous pressure to reduce time-to-market cycles. With more and more departments utilizing IT and undergoing “digital transformation”, it is becoming difficult to standardize test management and accelerate test execution. However, comprehensive testing across business-critical applications involving change and process is always needed. Testing should never be a bottleneck for releasing change in any enterprise. Enterprises are now slowly moving to test automation to ensure continuous delivery of change and quality.
However, to meet quality standards when implementing continuous change, enterprises need a single tool which can not only ensure test automation, but also take care of test management, test execution, test tracking, and test collaboration. After all, choosing the right testing solution can remove the problems involved in testing, ensure stress-free testing for test managers, and help enterprises stay ahead of the curve. So, what is the right testing solution for your enterprise? You’ll get the answer at the end of this E-Book.
What is SAP S/4HANA Testing?
SAP S/4HANA testing is the process of verifying the functioning of SAP S/4HANA with an intent to find out
whether the ERP system meets the specified requirements and ensure that the system is defect free. Testing
is essentially a quality assurance process that ensures the correct operation of SAP S/4HANA before the
project implementation Go-Live. Also, testing is not just restricted to the project implementation and project
go-live phase. SAP S/4HANA is a new-generation ERP with regular updates and new releases. So, testing is
necessary at each upgrade and therefore, customers must realize that SAP S/4HANA Testing is a never-
ending process.
What is the SAP S/4HANA Test Lifecycle?
SAP recommends Activate methodology for S/4HANA Projects. Each phase of Activate has its own set of buzz words associated with it. For Discover, it is Value Scoping and Impact Study. Similarly, for Prepare it is Project Team Preparation and for Explore the words associated are Cutover Preparation and Design of the system. Coming to the Realize phase, the keyword associated is Testing, Testing, and only Testing.
As per the SAP Activate Methodology, the test lifecycle consists of three phases:
1. Test Planning 2. Test Preparation and Execution 3. Test Validation
©KTern
3
E-Book – What you should know about SAP S/4HANA Testing Automation
Image 1: Testing Life Cycle as per SAP Activate
Test Planning
To minimize the number of issues during or after Go-Live, it is critical to manage the quality of the solution. As part of any scheduled maintenance event, it is necessary to consider and plan the testing cycles required to mitigate production support issues. In this stage the following key elements of the test planning must be documented:
1. Project Testing Objectives & Assumptions
➢ Unit Testing is complete before Integration Testing ➢ Unit Testing is only required for delta functionality
2. Test Scope 3. Types of Testing
➢ Unit Testing ➢ Business Process (String) Testing ➢ Integration testing ➢ Data Conversion Testing ➢ User Acceptance Testing
4. Testing Approach
➢ Description on how different test types relate to each other, e.g. how a successful unit test is a prerequisite for doing a string test or how a migration test results might lead into a pre-requisite for a user acceptance testing
5. Testing Deliverables 6. Test processes per phase, test environment, test tools
©KTern
4
E-Book – What you should know about SAP S/4HANA Testing Automation
7. Test Automation 8. Testing Tools - which tools will be used to perform different tests (e.g. KTern, SAP Solution
Manager) 9. Defect Management
➢ Description of how defects will be documented (e.g. Test Workbench in SAP Solution Manager)
10. Roles and Responsibilities
➢ Description of required test roles and responsibilities ➢ Test lead and responsibilities of individual project team members related to testing
Test Preparation and Execution
As determined in the evaluation of the existing test materials and documented within the testing plan, additional assets may need to be developed to support the execution of the testing cycles. Develop the missing test materials and test scripts in accordance with the detailed test plan.
Test Preparation
Within in each implemented solution scope, the following steps need to be executed:
1. Extend best-practice test cases 2. Develop delta process test cases 3. Finalize integration and user-acceptance test cases and plan 4. Prepare approval procedure 5. Prepare tool adaption and delta user acceptance test training 6. Prepare a test environment with the required test data as defined in the activity Test Planning. Once
the tests have been planned and test data is available on the test systems, testing can begin.
Test Execution
The typical process for the Realize phase is as follows:
• Software developers perform unit tests in the DEV system. Depending on the type and scope of the test cycle, various functional tests are performed.
• Manual testers are provided with the tester handout document and receive details regarding their test package by e-mail.
• Automated tests are scheduled or started directly.
• Every test that is executed is logged and documented with test notes and a test status is set manually or automatically.
• If the system responds in an unexpected way during manual testing, for example, if an error message appears, the tester records the incident in the corresponding ITSM system, attaching screenshots, log messages, and so on. Usually, this also must be done manually even for automated tests.
• The incident is sent to the persons responsible for analysis and categorization of defects, who then corrects the defect in the development system.
• The correction is transported to the test system according to the existing arrangements and timelines, where it is then retested.
©KTern
5
E-Book – What you should know about SAP S/4HANA Testing Automation
Given the complexity and heterogeneity of modern software solutions, SAP recommends performing Integration Validation, especially for important business processes. This involves gathering and evaluating a substantial amount of data from the software applications that are active while a given business process is being executed. This type of validation allows you to identify hidden warnings and error messages that frequently occur at the interfaces between applications. Furthermore, the operations team should monitor the testing system like it was a production system. This will help gain early visibility and hands-on experience for possible production issues which might arise in the future.
If large-scale changes are made or new software solutions are implemented, load tests should be performed before these are used in production. These tests simulate a situation in which the expected load (known number of users and background load in a load-peak situation) is simulated. In doing so, the system behavior while handling large data volumes can be inspected. Throughout the entire test cycle, test coordinators must monitor the test status and progress, as well as the processing status of incidents that have been reported. The quality of test data and test scripts directly affect the stability of the production system following the Go-Live of the change event. Consider an array of representative variances when preparing for the execution of the regression test cycle. It is important to execute realistic data sets that represent production operations of critical business processes.
Test Validation
The steps involved in the Test Validation Phase are:
1. Prepare User Acceptance Test plan: Update the existing integration test cases, end-to-end customer business process scenarios, based on the learnings from previous test phase. UA test plans and test case documentations are stored in Solution Manager.
2. Prepare and document User Acceptance Test Case #1 – n: The purpose of this task is to document the UA test case outlined in the UA test plan. This activity also contains aligned setup of relevant test data that will be commonly used.
3. Execute User Acceptance Test Case #1 – n: Perform the test according to previously defined plan. During the test all issues must be logged and documented in the system for traceability purposes.
4. Perform defect resolution for User Acceptance Test: Resolve any issue identified during testing. It is crucial that these issues are re-tested by the users that reported them (or designated re-testers) and that they are confirmed.
5. Obtain User Acceptance Test Sign Off: Obtain customer approval (sign-off) of the User Acceptance Test.
Testing is a time-consuming process, but make sure that you do not cut corners. Quality time spent on testing can go a long way in saving you from unforeseen glitches.
Problems in SAP S/4HANA Testing
In today’s digital world, SAP S/4HANA testing remains a great area of concern for both the business users and the business decision makers. The major difficulties encountered by the key persons during SAP S/4HANA testing are –
➢ Poor documentation of testers ➢ Low degree of test automation ➢ Maintaining test audits ➢ Cost benefit analysis of testing
©KTern
6
E-Book – What you should know about SAP S/4HANA Testing Automation
A Devil in Disguise – Testing Failures in SAP S/4HANA projects
SAP Landscapes are complex and business processes are coupled in, out, around, and beyond the standard functionalities. The disruption that S/4HANA brings technically, functionally, and business process wise to these landscapes is huge. This disruption requires careful compensation in the form of testing. However, there are times when testing proves to be a huge devil in disguise. These are the times when there are significant testing failures. Let’s look at some of the major reasons behind SAP S/4HANA testing failures and how they can be avoided.
Not understanding the “As-Is” and “To-Be” Landscape Architectures
Why should someone even think about landscape architectures while considering the move to S/4HANA? It is important to understand the current to-be process. SAP ECC to SAP S/4HANA migration is not just a technical upgrade. In fact, it relies a lot on data and processes. For all we know, the simplifications from S/4HANA cut across everything, right from field changes to process simplifications. Therefore, a vision of the landscape architecture design becomes inevitable during the test requirement phase.
Lack of knowledge about processes or the process itself
An organization requires a clearly defined Business Process Documentation to start the transformation project. But, most of the times, organizations miss out on this critical factor. They do not undertake this study before starting the project. This could pose a serious threat to the transformation project. Without the business process understanding, it would be nearly impossible to move into S/4HANA with so many simplifications.
Improper Scope Definition between testing and defect management teams
Proper end-to-end testing essentially implies that we are talking about a BPML (Business Process Master List) which has been running a business for many years. An average sized enterprise may have around 1000+ processes, each having multiple test cases based on their data. When a Responsibility Assignment Matrix (RACI Matrix) is drawn on the test management, it would make the system integrator accountable. The Project Health deteriorates severely during testing because of the involvement of the test engineering teams as well as business end users. So, a clear scope definition on the test items as well as the KPI Metrics is required. If not a plan, then at least a testing strategy is needed.
Piled up Conversations and Collaboration messages
The introduction of external business and testing teams brings along hundreds of email communications and issue logs every single day. A pile-up of communication data means improper test and defect management which would reduce the project health as well as the productivity of the consultants. A conversation pertaining to a test case log or an issue log can information like the criticality of the business process to be tested, type of defects, type of issues, status of defects, status of issues, aging of issues, and more.
Improper identification of stakeholders and criticality of businesses
A transformation project’s testing is not synonymous with 100% coverage of the BPML items. It could just cover the items that are getting impacted and are critical to the business. This way, the project team can focus on what’s necessary and get the test cycle completed quickly. This may give rise to two more questions:
©KTern
7
E-Book – What you should know about SAP S/4HANA Testing Automation
Question 1 - “Who knows what gets impacted when the move to S/4HANA is made?”
Question 2 - “Who can estimate the criticality of the business processes?”
Thus, identifying the key stakeholders and the criticality of the business is very essential for the success of the testing phase.
Improper execution of plans and phase signoffs
When we talk about test plans and cycles, it involves what must be executed, when it must be executed, how it must be executed, who must execute it, the entrance criteria definitions, and the exit criteria definitions. A lot of effort will be saved with the help of these information. Proper execution of test cycles will have a direct impact on the proper execution of project plans, phase closures and phase signoffs.
Improper Monitoring and Reporting Methods
With huge data being generated every day across multiple workstreams, reporting would be a challenge with data accumulated in multiple places and across spreadsheets. There are several chances to miss out on critical impact items and ending up with overhead issues in the long run.
As you can see, it’s easy to lose your way during SAP S/4HANA testing. Therefore, it becomes clearer as to why enterprises need a single tool which can not only ensure test automation, but also take care of test management, test execution, test tracking, and test collaboration.
5 Steps for S/4HANA Testing Automation
Step 1 – Identify critical business processes
Identifying the most critical business processes to be tested is the first step in scoping any test plan. This
requires business understanding and current data on business processes – derived from SAP systems and
delivered with actionable insights. This essentially means hard data from which we can infer the business
processes that are most critical or used most often. In addition, multiple variants exist in the way each
business process is fulfilled by different users. For scoping, statistics of business processes must include all
usage variants and the respective usage frequency.
For the sake of an example, let’s take the following scenarios. Generating a report comparing delivery times
and other parameters of different suppliers may not be test-worthy as it is only executed once a year when
the suppliers’ agreements are up for renewal. Other processes like issuing a discount for example, might
have variants, but only two of them are commonly used and therefore are worth testing.
You may now realize that the main scoping challenge is focusing effort and resources on business processes
that must be tested instead of wasting them on irrelevant or outdated processes. Unfortunately, the usual
test automation solutions do not address this challenge – mainly because they are unable to map the raw
business processes of SAP. Some organizations use excel or other spreadsheet software to keep track of
their business process data – but this is the absolute worst idea to follow for your testing strategy as they
must be manually maintained and act as counterproductive measure to automation.
Take-Away
A tool that can automatically generate business process data and present them as insights for test case
©KTern
8
E-Book – What you should know about SAP S/4HANA Testing Automation
creation, test execution, and test reporting is needed to identify the most business-critical processes to test.
This would help to save time, effort, and the number of resources needed.
Step 2 – Choose the right business processes to automate
Enterprise and especially testing teams should realize it is easy to get carried away. It is easy to get into a
rush and try to automate all business-critical processes. Take a chill pill. Findings state that most
organizations automate only 30-40% of the business-critical processes. This is why only 1/5th of the
regression test cases consist of critical business processes.
But how do you know which test cases to automate? What factors must one consider? Is there anything
such as useless automation? Is there a chance automation is causing more harm than good?
Businesses change often. This is the status-quo of the digital world. For example, a simple software purchase
in an organization might need to go through multiple approvals along the length and breadth of the
company depending on company policies. In addition, policy changes could change rapidly based on
management changes and many other factors. Automating and maintaining test cases for changes which
happen too often is not feasible.
Likewise, some business processes can have multiple roots and dependencies. For example, the shipping
vendor location may depend on geographic destinations, order cost, order channels, payment method,
taxation, and available shipping partners. Some dependencies may in turn represent many other business
processes. Creating and maintaining automated test cases does not make sense in this case either.
So then, what is a good example of an ideal test case for automation, you ask? Test cases for automation
should be lengthy, repetitive, and effort-intensive business processes. “Order creation” is one good example.
Order to cash, Procure to Pay, Record to Report, Plan to Produce, Plan to Maintain, Fund Management are
all good example for ideal test cases for automation.
Take-Away
When deciding on the test cases to automate, it is important to only automate the lengthy, static, labor-
intensive test cases. Test cases which are dynamic or have multiple dependencies should be kept manual. A
tool (like KTern) which utilizes Machine Learning and RPA (at KTern we call it i-RPA) can help automate test
cases for the relevant business processes and eliminate the need for manually engineered test cases.
Step 3 – Capture the knowledge of key users
SAP is vast. SAP is old. SAP is filled with knowledge for those who search for them. Experience and all the
“dog-years” worth of SAP knowledge are highly important. Testing acumen is important. But, so is business
process know-how. In fact, Capgemini World Quality Report lists ‘business process understanding’ as the key
skill for Testing and Agile Quality Analysis according to 64% of the IT leaders whom they surveyed.
Identifying the right users, contacting them, and obtaining their application knowledge is a laborious activity
and is time and effort intensive. The key users prioritize their individual business objectives and it could be
hard to get the right inputs from them. Incentives could help them to devote time and get their input of
recommended test scripts. Alternatively, a functional analyst maybe needed to record and monitor usage
data to pick the key users and get their inputs. If the employees are present in multiple locations across the
globe, then the inputs of key users in each location is needed since there could be process variants from one
location to another.
It might be clear why capturing and maintaining business knowledge is difficult. Some organizations use
©KTern
9
E-Book – What you should know about SAP S/4HANA Testing Automation
Excel sheets or other spreadsheet software – but this is not recommended. It is time consuming to edit,
share, and keep the sheets up-to-date from different key users across the globe.
Take-Away
Identifying the right key users to prepare test scripts, following up with them to get the information,
updating test scripts, and confirming the validity of test scripts are all difficult processes – especially since
the production system is constantly changing. The right tool (like KTern) will capture the right test cases
along with the key users “know-hows” to resemble production and usage patterns. Provide instant
information and real-time insights by removing batch jobs and manual effort.
Step 4 – Preparing test cases which incorporate business process variants
Business processes are not constant. They vary from department to department, function to function, or
location to location. For example, a business user in Italy may complete the same business process
differently from that completed by a business user in the United States. This may be attributed to different
locations filling the system fields using different time and date formats, different metric systems, or other
factors. This is where the coverage of automated test cases plas a major role. The automated test cases
should cover the frequently used variants of the relevant business cases.
Take-Away
The right test automation tool will generate test cases which incorporate the different variants of relevant
business processes based on actual usage and up-to-date data from production.
Step 5 – Test what must be tested
Perhaps the most important part of automated testing is test case maintenance. Maintaining an accurate
test case repository can result in critical savings in test case management. It is therefore important to
maintain updated data sources which are current on usage patterns and variants. A dynamic record of the
change statistics of business processes helps to eliminate costs and produce meaningful savings for test case
maintenance.
However, this seems to be easier said than done. Studies and surveys indicate that the biggest challenge for
testing leaders is to maintain and update their test case library to resemble the scenarios in production.
Take-Away
The right test automation tool will help maintain an updated test repository and captures the business
knowledge of the relevant test cases in order to maintain a near replica of production.
Nice To Haves along with Testing Automation
Test automation is just the first step. But, a good testing tool will be able to offer much more. Some of the
key features which are nice to have along with test automation are workflow management, defect
management, and an easy-to-use UI and UX to accelerate and increase user adoption.
Workflow Management
Workflow management is a really good nice to have. Coupled with other process orchestration activities it
can reduce a significant amount (up to 35%) of the testing effort needed. Some of the features which can
help are sequencing of manual tests, handover of testing tasks between key users, progress monitoring, and
©KTern
10
E-Book – What you should know about SAP S/4HANA Testing Automation
a central defect management center to speed up the complete test execution. Additional features which can
help are “Time to Test Notifications” when a user has to test within a business process, “Retest
Notifications” to testers for defect resolutions and “Close Notifications” when retesting is a success.
Key users can benefit a lot from the features of workflow management. They can execute their testing steps
without any dependencies – even if the business process requires multiple steps and multiple testers.
Defect Management
Automation can also help with managing defects and resolving them. A good defect management tool will
provide a central center for monitoring defects, their resolution, and a view of the business processes
affected by the defects. In addition, the right defect management tool and features can help to
automatically identify all the other tests which could be affected by the defect and warn the testers and the
test manager until the main defect is resolved.
User Adoption
People are always busy. The same applies for the key business users. They could feel it to be a severe
learning process to try and adapt to yet another new tool. It is therefore important to have an intuitive user
experience to promote user adoption. The application must be simple as logging into your email inbox, and
require training as minimal as adjusting to a new mobile phone.
KTern Features for S/4HANA Testing Automation
8.1 Test Case Identification
KTern enables test scripts to be loaded by connecting KTern with an ECC system and running “Check Test
Case Recommendations”. The Results are loaded in a hierarchical manner as Plan, Scenario, and Test case.
Each Plan can have “n” scenarios under it and in turn contain “n” test cases. Recommendations are provided
based on Process Configurations and Customizations enabled in the system. Test Cases are recommended
based on documents generated and business criticality.
8.2 Test Case Execution
The typical process for the Realize phase is as follows:
• Software developers perform unit tests in the DEV system. Depending on the type and scope of the test cycle, various functional tests are performed.
• Manual testers are provided with the tester handout document and receive details regarding their test package by e-mail.
• Automated tests are scheduled or started directly.
• Every test that is executed is logged and documented with test notes and a test status is set manually or automatically.
• If a system responds in an unexpected way during manual testing, for example, if an error message appears, the tester records the incident in the corresponding ITSM system, attaching screenshots, log messages, and so on. Usually, this also must be done manually even for automated tests.
• The incident is sent to the persons responsible for analysis and categorization of defects, who then corrects the defect in the development system.
• The correction is transported to the test system according to the existing arrangements and timelines, where it is then retested.
©KTern
11
E-Book – What you should know about SAP S/4HANA Testing Automation
In Image: Test Suite Catalog
In Image: Test Center
©KTern
12
E-Book – What you should know about SAP S/4HANA Testing Automation
In Image: Test Lab
In Image: Auto Test Console
KTern has an inbuilt test management framework to automate and manage test executions and allows multiple stakeholders to be involved in the test cases. Test Execution involves multiple executions in different systems during different times of the project. KTern enables users to collaborate and record discussions against each test item. The User Management framework can be used to invite business users, test engineers, IT SMEs, and consultants to be part of the project. In addition, Assignment, Workflows, and Notifications are enabled by the KTern Management framework.
©KTern
13
E-Book – What you should know about SAP S/4HANA Testing Automation
In Image: Defect Repository
The Test Suite Framework would facilitate creating test scripts, recording test scripts, and recording test
executions against a test case. Defects generated from test cases are managed using the Defects Repository
to help manage Test Script Documentation, Test Execution Logs, and Defects created. KTern and its
intelligent RPA (i-RPA) framework help execute test cases on the fly through the application layer. This helps
reduce the effort and time needed to unit test all test cases. In addition, the KTern i-RPA framework
facilitates “record and play” to help users record test cases and execute them for multiple data sets. Defects
for failed test cases are auto-created by KTern based on message text type received.
In Image: Test Case Reporting
©KTern
14
E-Book – What you should know about SAP S/4HANA Testing Automation
8.3 Test Case Reporting
KTern and its versatile analytical framework allow test case reporting in a robust manner. Test cases are
sliced and diced based on different KPI models and provides an accurate track on the amount of test cases,
executions failed, test cycle progress, defect status, resource utilizations, test plan coverage etc.
8.4 VVV Scoring (Value, Volume and Velocity Analysis)
VVV Scoring covers the Volume, Value, and Velocity Analysis of the business per transaction usage and helps
decide the critical business processes which must be tested. As you may recall, identifying the most critical
business processes to be tested is the first step in scoping any test plan. This requires business
understanding and current data on business processes – derived from SAP systems and delivered with
actionable insights. KTern is a tool that can automatically generate business process data and present them
as insights for test case creation, test execution, and test reporting is needed to identify the most business-
critical processes to test. This would help to save time, effort, and the number of resources needed.
In Image: VVV Scoring
8.5 Sign Off Wizard
Sign Off Wizard provides a hierarchical view of the Test Cycle Approvals with business stakeholders. It
provides an unimagined level of transparency to the testing process and makes it easy to complete phase
closures and signoffs during the testing phase.
©KTern
15
E-Book – What you should know about SAP S/4HANA Testing Automation
In Image: Test Suite Sign-Off Wizard
Next Steps
▪ Visit our website.
▪ View other whitepapers and other resources.
▪ Get Started with KTern and explore the free KTern Demo or Sign-Up for your free Trial.
▪ Get in touch with us so that we can set up a call and discuss further.
About KTern
KTern is an SAP Certified product to automate and manage SAP S/4HANA migrations from start to finish. With 12 inbuilt modules, the AI & RPA driven "S/4HANAPEDIA" Engine, and the vast knowledge base of SAP and SAP Partners, KTern will reduce up to 1/3rd of the total migration cost and timeline. Given below are some of the salient features of KTern across the different stages of an S/4HANA Transition. Before the Transition
• Automated assessment to analyze technical, functional, & business impact of moving to S/4HANA
• Auto-recommendations pertaining to testing strategy, landscape & infra, and data archiving
• Auto-generation of an S/4HANA conversion blueprint incorporating the right plan, the right effort,
the right cost, the right system sizing, the right project management, and the right quality
During the Transition
• Custom process refinement using preset automation rulesets for custom code remediation
• Business Process Automation Testing based on ECC system usage and inbuilt datasets
• Automated Data Validation based on the migrated data from ECC to S/4HANA
After the Transition
• Provision of a repository of test cases, issues, and bugs for faster and safer future upgrades
• Readily available and accessible data for audit requirements
• UX transformation through Fiori App Recommendations