SAP Testing Tools Comparison: Key Considerations

© 2017 TricentisAll rights reserved.

Companies relying on SAP must continually test their systems to avoidbusiness disruptions stemming from SAP upgrades, service packs,customizations, and enhancement packages. Failure to detect or address achange could bring critical business processes to a standstill.

Yet, much of the testing responsibility falls on key users, who are expected tovalidate transactions alongside all their other duties. At the same time, thecomplexity of what needs to be tested is increasing, with end-to-end businessprocesses commonly extending beyond SAP and into APIs, web UIs, mobileinterfaces, custom back-end systems, packaged applications, and more. Howcan you effectively manage change in light of these challenges?

Leading enterprises have found that SAP test automation tools are critical fordetecting and mitigating impacts to core end-to-end business transactions. Tohelp you choose the SAP testing tool that best suits your organization’sspecific needs, this paper outlines some key points to consider as youevaluate and compare tools.

2

With inadequate or laborious SAP automation technology, yourtest automation initiative will be an uphill battle from the start. Itwill also likely be a losing battle if it’s easier to manually testcertain elements than to create and maintain automated testsfor them. Consider:

• How much manual effort is actually involved in defining theautomated tests? In updating them?

• What SAP elements can be automated out of the box? Forexample, does the tool provide out-of-the-box support for:

• Unlimited navigation in SAP tables via businessreadable content identification

• Verification of values in result tables (technically:label containers)

• Status checks in SAP tables

• Navigation in SAP trees via business instruction

• Creation and verification of IDocs

• The various SAP Fiori elements associated withyour processes

• How are dynamic SAP Control-IDs handled?

• In what situations is coding required? How frequently willthe associated coding need to be updated?

• Can tests be automatically generated from logs? Recordinguser behavior? Drag and drop from a library of options?

• Is the interface intuitive? How long does it take end users toget up to speed and recognize the tool’s value?

• How much manual effort is required to reuse test blocks(e.g., for an SAP login process)?

3

Ease of SAP Automation----------------------------------------------------------------------------------

---------------------------------------------------------------------------

Automated generation of SAP tests from production data hasthe potential to make business process testing significantlyfaster and easier. However, it’s important to consider exactlywhat steps are automated—and how intelligently. Consider:

• How does the tool generate a list of processes to test? Doesthat list have an item for each and every transactiondetected, or does it analyze and correlate the data todeliver a master superset of transactions?

• Does it prioritize the transactions and help you determinewhich are most important to test?

• Does it actually generate tests that cover the most criticaltransactions? How much manual intervention is required tocreate these automated tests?

• How are the tests updated as usage patterns change?

• How rapidly does the resulting test suite execute?

4

AutonomousSAP Testing

Without easy, intuitive SAP automation, your business process testing initiative will be dead on arrival. However, the ability to extendbeyond SAP and cover every technology involved in your end-to-end transactions can make the difference between fragmentedspurts of automation and a comprehensive, trusted test automation strategy. Consider:

• If your core business transactions involve technologies such as Java, mobile front-ends, SOAP or RESTful APIs, etc., how easily andeffectively can the tool test those elements?

• Can the tool automatically test the integrity of any BI data that your business relies on?

• Can a business user feasibly automate end-to-end tests that include non-SAP technologies such as mobile, APIs, etc.? How muchadditional training would be required?

• How quickly does the vendor introduce support for new technologies?

• Can values from one type of test be used to parameterize other types? For example, can the result of an SAP field be used topopulate an API test?

• Can you use a central test data management and generation interface for all your tests, or do you need to have separatesolutions for SAP and non-SAP elements?

5

Technology Support Beyond SAP

Business analysts are burdened with testing in addition to all their otherduties. Resources for testing are limited, so it’s critical to understand andfocus testing efforts on the top business risks. Consider:

• Does the tool help you prioritize testing based on the risk associated withyour requirements/user stories?

• Does the tool help you determine what tests and test data are requiredto cover your top business risks?

• Does it tell you what level of business risk coverage your tests areachieving?

• Does it report test results in terms of your requirements and risks?

• Does it help you determine the minimal number of tests needed toexercise all application logic?

6

Risk-Based Testing

The increasing pace and scope of SAP updates requires comprehensive,near-continuous testing to identify and resolve potential impacts. You notonly need to know what changed; you also need to ensure that your testsuite is efficient. Otherwise, you might waste time continuously updatingtests that do not add value. Consider:

• Does the tool help you identify and update tests impacted by changedSAP transports—including end-to-end test cases that extend beyondSAP?

• Does it help you generate dynamic TBOMs?

• Does it tie the results of the impact analysis to risk-weightedrequirements (from Solution Manager) to help you prioritize updating?

• Does it help you continually optimize test suites so you don’t wastetime updating “no value added” tests each time a transport changes?

7

Impact Analysis

Many organizations create automated tests, but few achieve sustainable test automation rates. Often, falsepositives, timeouts, and incomplete tests erode trust in the results. Also, the test suite grows too unwieldy toupdate—to the point where it seems easier to test manually than update everything that’s out of synch.Consider:

• How easy is it to update tests if an element is added, removed, renamed, etc.?

• How easy is it to update test data used across the transaction (SAP and non-SAP elements)?

• Does the tool help you reduce test case redundancy to prevent bloated, difficult-to maintain test suites?

• Does the tool ever create duplicate modules when tests are created? If so, how are they resolved?

• Can you execute tests even if dependent components are unavailable or in an unstable state?

8

Test Maintenance and Optimization

Studies report that 81% of organizations are adopting DevOps and 91% are adopting Agile. However,you’d have to search long and hard to find an enterprise that doesn’t have scaling Agile and adoptingDevOps on its radar. It’s not a matter of if testing will need to accommodate DevOps and Agile, but when.Consider:

• Can you integrate the tool into a “best of breed” DevOps pipeline with a variety of CI/CD tools?

• Does it offer bidirectional synchronization of requirements, issues, and tests with popular tools suchas JIRA, Rally, GitHub, etc.?

• What capabilities does it offer that help you ensure rapid, successful execution of unattended tests?

• Does it offer service virtualization to ensure that end-to-end tests can be executed during CI—withoutdependency issues causing timeouts, incomplete tests, false positives, and/or inaccurate results?

• Does it help you prioritize a condensed set of “smoke” tests suitable for CI as well as a larger set ofnightly tests that provide broader risk coverage? Are there ways to achieve the desired test executionspeed (e.g., with distributed execution)?

• Does it enable “shift left” defect detection techniques such as exploratory testing?

• Does it allow you start testing even when dependent components (e.g., third-party applications, APIs,etc.) are not yet completed or are evolving in parallel?

9

Readiness for Agile and DevOps

In addition to your own research, youmight want to consider what analysts haveto say. Each year, Gartner evaluates thetop tools that met their criteria forenterprise-grade test automation tools.The evaluation involves inquiries withGartner clients, surveys of tool users,vendor responses to Gartner questions,and product demonstrations. This year,the results were as follows:

10

Analyst Evaluations

For the detailed analysis, including thepros and cons of each tool, you candownload the latest report from theTricentis web site. You can also review asynopsis of the previous year’s rankingson TechBeacon.

Stay in touchwww.tricentis.com

LocationsContact

Take the fastest and easiestpath to automated businessprocess testingLearn about Tricentis SAP Testing—zero-touch, scriptless testing, industry-leading SAP testautomation & more