a new approach to automated web testing.7-2007.final

SOASTA Concerto™ A New Approach to Automated Web Testing

White Paper

©July, 2007, SOASTA, Inc. All rights reserved. The names of actual companies and products mentioned herein may be the trademarks of their respective companies. This document is for informational purposes only. SOASTA makes no warranties, express or implied, as to the information contained within this document.

SOASTA Concerto™ A New Approach To Automated Web Testing 1

Introduction

The world is changing. Everything in enterprise software is moving to the Web. The economic advantages of browser software delivery (no installs, instant availability everywhere, updates applied centrally) create great demand for this type of enterprise application. But, these new Web applications are difficult and expensive to test. In fact, most Web testing is currently hand built.

While there are many point testing tools on the market today, they are too hard to use and are breathtakingly expensive. With the shift to SOA and Web 2.0, the industry requires an integrated, collaborative, affordable environment that allows users to quickly and easily test Web applications and services. SOA will fail if a new testing paradigm does not arise that can handle the complexity of real-world Web testing scenarios. What do we mean by real world? Web applications and services made up of thousands of messages and events, supporting tens of thousands of users, that are based upon a variety of protocols…SOAP, REST, HTTP, Ajax and so on. This is the problem that SOASTA was founded to solve. SOASTA is delivering the industry’s first browser-based Web testing service. SOASTA Concerto is Web native and purpose built to support the testing of any Web application or service…from the message layer to the user interface. SOASTA simplifies today’s complex world of software testing by substantially reducing the effort and technical expertise necessary to test Web applications and services. This will make Web testing more productive, while enabling companies to deliver higher quality software to market more quickly. While there are many acronyms that have been invented to describe the different variations in approach, such as SOA, and Web 2.0, there is deep consensus that XML messaging and Web Services are the basic building blocks of these new Web applications. Proper and complete Web testing is difficult because:

• The architecture of a Web application or Web service involves more technology (browser,

Web server, application server, database server, etc.) than in a client/server architecture. While it is a better, less expensive, and easier delivery platform for end users, it is also more complex and offers many more technology choices for the developer. These technologies must also be handled by the tester as well. This means that the testing task is bigger and more difficult overall.

• Web standards. We have many standards. We also have many things that are not standards but operate like standards (i.e. HTTP, HTML, SOAP, XML, REST, XML-RPC, Ajax, JSON, CSS, etc.). This diversity gives testers a wide universe of protocols to test. One might argue that testers can standardize on a smaller set of technologies, but one of the major drivers of Web applications and services is the ability to integrate an organization’s applications with those of customers, partners, and suppliers. In doing so, however, the testing scenario becomes even larger and more complex.

• The Web is not easy to test. There are no standard connectors to testing environments.

• There are many types of testing:

- Functional/regression testing (does my application work?) - Performance testing (is my application fast enough?) - Load testing (how many users can I support with decent performance?)


- Stability testing (how long will my application run without a reboot?) - Platform testing (will my application work across operating systems?) - Web UI / Ajax testing (will my application work across all browsers?)

This new Web-oriented approach provides enterprises with a flexible and powerful architecture that promises better, easier scalability, while preserving investments in existing software. Further benefits of this approach include code reuse, business agility and the technical ability to launch processes “ON DEMAND”. Every leading software application vendor (SAP, Oracle, salesforce.com, etc.) is moving rapidly to deliver versions of their products that expose web services to support this approach. The analyst community is unanimous in supporting this architectural movement and is forecasting its success in the Automated Software Quality (ASQ) category over the next five years:

• One predicts that, “By 2008, SOA will provide the basis for 80 percent of new development projects.”

• One states, “Quality and software testing, a longtime stepchild of the software market, will get

hot. This will be especially true in the context of ensuring that business services perform the functions they are designed for in a predictable manner.”

• Another reports, “Although SOA once was used primarily by large enterprises to lower

integration costs by using web services, it is now reaching broader implementation in the United States, regardless of company size or vertical industry designation.”

• A third analyst states, “SOA is as big a transition as the movement from host-based

architectures to client/server and from client/server to Web applications. Each of these generations has required different approaches.”

• A fourth analyst indicates, “The continual shift toward service-oriented architecture will drive

demand for new ASQ capabilities to assess and certify Web services and components.”

• A fifth analyst remarks, “There is still a real lack of solutions for doing any real testing of Web services in the context of SOA” and, “. . . for SOA implementations, testing is essentially being done in a custom way.”

While these new application architectures deliver the ability to quickly integrate and customize applications, they are necessarily complex. A full SOA implementation of an application system in the enterprise will expose all of its transaction types as operations. These operations are then invoked by sending the correct message. A complex application like ERP can support tens of thousands of message types. This complexity makes the task of testing and certifying applications far more difficult than in the past. SOASTA believes there is a large market requirement for easy-to-use integrated testing solution that simplifies and drastically reduces the effort necessary to build, modify, demonstrate, test, certify, and re-certify Web services and applications. In 2006, SOASTA will implement and bring to market a suite of visual software products that directly and easily support the creation and playback of complex message streams.

Traditional Testing Approaches An examination of the marketplace is necessary to understand why current solutions do not adequately meet the testing requirements of new Web applications and services. There are three product approaches to software/services testing.


• GUI Testing Tools • Capture/Replay Testing Tools • Perl Scripts/Hand-Coded XML/In-House Tools

GUI Testing Tools This is the largest category of existing ASQ tools representing over 80 percent of the current market. Mercury Interactive (now part of Hewlett-Packard) pioneered this approach to automated testing in 1991 when it introduced Windows technology to “capture user actions” in the user interface and automatically generate reusable scripts. Those scripts are then replayed in WinRunner™ to test Windows UIs and in LoadRunner™ to simulate hundreds and thousands of simultaneous Windows end users. This was a revolutionary product approach at the time because it allowed business analysts and end users (instead of software engineers) to participate in creating and running tests to certify Windows-based client/server applications. Tests could be “captured” and reused electronically. IBM Rational’s TestStudio™ product is very similar to Mercury’s. These are great products for GUI Testing of client/server applications. They do not address the needs of developers, sales engineers, business analysts and testers to create message scenarios for testing complex, composite web applications. Capture/Replay Testing Tools A very popular product approach is to capture messages from the existing application and playback those messages for regression testing. SOASTA believes that this capture capability is one necessary feature in a full solution to testing a message-based application. While capturing messages is the easiest and fastest way to generate message volume, this approach does not ensure the completeness and validity of the captured test. In addition, messages cannot be captured while an application is under construction. SOASTA Concerto supports message capture, creation, and generation. Perl Scripts/Hand-Coded XML/In-House Tools Testing, certifying, and demonstrating service-based applications is time consuming, complex, tedious, error-prone, and expensive. Existing ASQ tools are NOT addressing this problem, and as a result, the testing of service-based applications is predominantly done using hand-coded messages that are “played” using Perl scripts or in-house tools. SOASTA’s conversations with industry practitioners confirm this and SOASTA believes that this is by far the current market-leading approach.


SOASTA Concerto™ SOASTA Concerto is the industry's first browser-based service for automating Web testing. With five powerful visual editors working as one, SOASTA Concerto’s seamless integration of test design, editing, and output offers everything users need to test and deliver high quality Web applications and services. SOASTA’s design combines a drag-and-drop visual interface with the ability to access the underlying message complexity. Through an easy and powerful UI, users create test scenarios that generate complex, parallel message streams. Task specific visual editors support the easy and efficient creation of targets, messages, and test compositions. Based upon the premise that sending a message to a target is the same as playing a musical note, SOASTA Concerto is modeled after today’s popular digital media creation tools such as Apple’s GarageBand and iMovie or Adobe’s Premiere. Building and executing a test or an event is as easy as creating and editing a piece of music or a movie from a series of film clips. This new metaphor requires a new taxonomy. As such, SOASTA Concerto has adopted the following naming convention:

• Message – raw XML or text for the protocol being used in the test • Test Clip – equivalent to a unit test

Test Composition – a combination of test clips arranged on “tracks” to make up a complete test scenario

• Playlist – multiple test compositions to be played all at once or one after another. Test compositions are “played” on a multi-track, timeline-based mixing board. Test results are captured, compared to expected results, and then certified or rejected. Test results are available in real-time and historically. SOASTA Concerto’s repository automatically maintains all messages, test compositions, and other system objects as well as a complete historical archive of test results. All tests are reusable. SOASTA Concerto supports the following types of testing:

• Functional Testing Modern Web applications and services contain multiple levels of technical complexity that makes building and executing valid Web application test suites more difficult than ever before. SOASTA Concerto handles this complexity, substantially improving testing productivity. SOASTA Concerto has built in support for testing Web applications and services using SOAP, REST, XML-RPC, POX as well as support for testing at the Web server using HTTP messages. Cross browser and Ajax testing completes SOASTA’s ability to test the complete Web application technology stack at every layer (separately and simultaneously). SOASTA provides a single integrated testing service for all levels of Web testing.

SOASTA’s visual approach supports the drag-and-drop creation of unit tests and is self-documenting. These unit tests may be reused many times as part of test compositions to simulate very complex testing scenarios. SOASTA’s unique Results Viewer allows the test designer instant access to errors in test results. No more digging through text logs to decipher the source of errors. Productivity is enhanced and tester fatigue is reduced.


• Load/Capacity Testing SOASTA Concerto has a number of features to support load testing at any point in the development cycle. Any number of test compositions can be run (or played) in parallel in SOASTA Concerto’s multi-track timeline, allowing for the precise control of timing and testing volume. No more 1 ... 2 ... 3 ... ENTER. Testers can run many parallel tests with one click of the PLAY button. Virtual tracks and virtual test clip support allow testers to quickly multiply the number of virtual users in a test. This is a very fast, easy way to create high-volume real-world load test scenarios. SOASTA’s Testing Analytics provide load charting that quickly summarizes load test results, combining performance statistics from both the test client and Web application server.

• Performance Testing

In many real-world load test scenarios performance issues are identified. This requires that developers and testers dig into the behavior of an individual component. SOASTA Concerto allows testers to quickly and easily create tests that stress specific components in a Web application or service. Tests can be repeated continuously and performance statistics are easily captured from the target application or service. Whether testers need to target the browser, the Web server, the Web service, or every layer at once, SOASTA Concerto supports the fast, easy creation of specific unit tests.

• Web UI / Ajax Testing

Not that long ago it was good enough to focus on Internet Explorer (IE) as the browser for a Web application or service. Today’s rich Browser UIs must run across IE, Firefox, Safari, and Opera. Ajax user interface approaches add another dimension that must be tested and certified. Browser actions and browser messages must be captured, simulated, validated, and certified across all commonly used browsers. Performance data and load testing becomes extremely important. UI testers need to validate the performance of many Ajax UIs running at once, as

well as the performance of the Web UI when many Ajax messages arrive in a short time. SOASTA Concerto uniquely supports for all Web UI/Ajax testing requirements.


Complexity is a huge issue in testing and certifying service-based applications. In a scenario involving an enterprise application (SAP, Siebel, Oracle, Peoplesoft, etc.), the number of operations available through the many web services can number in the tens of thousands. Even more daunting, a complete functional regression test intended for certification of that application will contain hundreds of thousands of messages. SOASTA Concerto provides the capability to organize and manage this innate complexity through its support of versioning and the ability to create hierarchical taxonomies of test cases. SOASTA Concerto is the best and easiest way to test Web applications and services. The products deliver unparalleled ease-of-use and productivity through an intuitive and unique visual interface. This allows complex message streams to be composed by any developer, sales engineer, business analyst, or tester. Enterprises are now able to create, test, and certify services-based applications in days rather than months.


Product Tour

SOASTA Concerto is available as a hosted service, installed on customers’ in-house servers as a subscription, or as an appliance. All interaction between users and SOASTA occurs over secure connections to the SOASTA online service. SOASTA requires that every user have a unique user-id and password.


SOASTA Central provides access to all SOASTA Concerto features and components. The objective is to allow the user to easily navigate and manipulate objects while supporting fast drill down to the message detail of a test.


The SOASTA Target Editor allows users to create a connection to the destination Web application or service to be tested. Targets are defined items in the SOASTA Repository that contain a specification of the information needed in order to properly format messages to send to that type of external application or system. A target location is the definition of a particular instance of a target. It contains a complete, concrete specification of the physical location of an implementation of the target so that messages can actually be sent to the target. Any number of target locations may be defined for each target.


The SOASTA Clip Editor is where users arrange and organize messages within a message clip. Similar to PowerPoint’s slide sorter, the SOASTA Clip Editor supports drag-and-drop and cut and paste for easy manipulation of individual messages and groups of messages. New messages may be added, deleted, and duplicated. Timing constraints may also be added in this editor.


The SOASTA Message Editor enables users to directly edit the data contents of individual test messages. Users are provided the flexibility of doing so at the message level, the results level, or at the core XML level.


The SOASTA Composition Editor is a timeline-based mixing board where testers play final test compositions. Test clips are assembled on the timeline to control sequence, tempo, and timing. This approach to building a complex message stream is similar to UI approaches found in professional media creation tools like Apple’s Final Cut Studio™ and GarageBand™.


The Results Viewer enables users to easily view and navigate test results by flipping through a jukebox style graphic. Errors are clearly marked in red as “failures” and users can drill down to the exact message or messages that failed and get detail on what went wrong.


Real World Example

The following is a real world, step-by-step scenario for creating a functional test against the Salesforce.com SOAP-based Web service. Your web testing is sure to be more complicated than this example. Step ➊ - Create a target called Salesforce.com Enterprise API v7 (na4) in the SOASTA Concerto Target Editor. This SOAP-based target is automatically created from the Web service description.


Step ➋ - Create test clips in the SOASTA Concerto Clip Editor to test specific Salesforce.com functional scenarios. An example of an “Account Creation” functional test is shown below. This test was created in SOASTA Concerto’s powerful Ajax interface allowing for easy drag-and-drop to quickly create tests.


Step ➌ - Create a composition in the SOASTA Concerto Composition Editor by dragging and dropping the test clips onto the multi-track timeline. The example below shows a test composition with seven different test clips. The first test clip (highlighted in green) logs into the Salesforce.com service and acquires the necessary login credentials. The remaining test clips all use these credentials to interact with the service and test a specific function. Having these test clips on different tracks in the timeline allows multiple user scenarios to be executed in parallel simultaneously.


Step ➍ - Play the test composition. Users can easily execute the test composition by merely clicking on the play button located on the timeline’s control bar. The orange playhead moves across the test composition to indicate which part of the test composition is being executed at any given time.


Step - Review test results. After the test composition is complete, the results can be reviewed in the SOASTA Concerto Results Viewer. The Results Viewer makes it very easy to review test results and quickly determine what errors (if any) occurred in the test. Once these errors are pinpointed they can be resolved quickly.


Summary Enterprise software applications are moving to the Web. Based on services and messaging, the objective is to make the transaction assets of an enterprise available as web services. Orchestration tools allow these services to be integrated into new processes within days. This new software generation delivers on the promise of the Agile Enterprise, but it is far more complex and has more moving parts than previous software generations. As a result, testing service-based applications is also more complex and requires a new approach. SOASTA’s vision is to provide a complete solution to automated web testing that addresses the unique requirements of this new generation.

About SOASTA, Inc. SOASTA, (pronounced sō'-sta), is a privately-held technology company focused on developing visual software products for automating the testing, certification and demonstration of web services and applications—for use by any developer, sales engineer, business analyst or tester. SOASTA is headquartered in Mountain View, California. To learn more about SOASTA and its product offerings, please write to [email protected] or visit us on the web at http://www.soasta.com. © 2007, SOASTA, Inc. SOASTA is a trademark of SOASTA, Inc. All other trademarks are the property of their respective owners. Contact: Lisa Bergamo SOASTA, Inc. 650.210.4942 [email protected]

SOASTA, Inc. 1975 Landings Drive

Mountain View, CA 94043 866.344.8766

http://soasta.com

a new approach to automated web testing.7-2007.final

Documents