combining automated functional and load testing

IntegratingRanorex and NeoLoad

Time to make peace between functional and non-functional testing

Welcome to the Ranorex-Neotys Webinar

Christian Breitwieser, RanorexInnovation Manager

Henrik Rexed, NeotysTechnical Evangelist

Agenda• Quick Company and Tool Intro• Performance Matters in User Experience• Benefits of Automated Functional Testing• Benefits of Combining Automated Functional and Load

Testing• Live Demonstration

• Functional test automation for desktop, web and mobile applications

• Cross-technology end-to-end testing• Tool for testers and developers

• Functional test automation for desktop, web and mobile apps

• Cross-technology end-to-end testing

Tool for testers and developers

Code Module

Action Based Modules

Ranorex Studio

• Deliver faster applications faster through continuous performance validation

- Who We Are

- Who We AreMission: Deliver faster applications faster through continuous performance validation

- Who We AreMission: Deliver faster applications faster through continuous performance validation

User Demands Process Revolutions

We’re Driven by Your Challenges

Technology Transitions

User Demands Process Revolutions

We’re Driven by Your Challenges Technology Transitions

User Demands Process Revolutions

We’re Driven by Your Challenges Technology Transitions

User Demands Process Revolutions

We’re Driven by Your Challenges Technology Transitions

Docker

Load and Performance Testing

24x7 Performance Monitoring

Continuous Performance Validation Solution

Load and Performance Testing

24x7 Performance Monitoring

Continuous Performance Validation Solution

Continuous Performance Validation Solution

DevelopmentContinuous

Integration & Component Testing

Continuous Performance Validation Solution

DevelopmentContinuous

Integration & Component Testing

Continuous Performance Validation Solution

Pre-productionSystem-wide

Testing

DevelopmentContinuous

Integration & Component Testing

Continuous Performance Validation Solution

Pre-productionSystem-wide

Testing

DevelopmentContinuous

Integration & Component Testing

Continuous Performance Validation Solution

Pre-productionSystem-wide

Testing

DevelopmentContinuous

Integration & Component Testing

Continuous Performance Validation Solution

Pre-productionSystem-wide

Testing

Production

Proactive Monitoring

DevelopmentContinuous

Integration & Component Testing

Continuous Performance Validation Solution

Pre-productionSystem-wide

Testing

Production

Proactive Monitoring

DevelopmentContinuous

Integration & Component Testing

Continuous Performance Validation Solution

Pre-productionSystem-wide

Testing

Production

Proactive Monitoring

DevelopmentContinuous

Integration & Component Testing

Agenda• Quick Company and Tool Intro• Performance Matters in User Experience• Benefits of Automated Functional Testing• Benefits of Combining Automated Functional and Load

Testing• Live Demonstration

User Experience is Crucial• 57% of users use their mobile devices to access

applications

• 40% of users move towards a competitor after a bad user experience

• 44-61% of users share their feelings on social networks

What isUser Experience? Usability

Ergonomics

Performance

Accessibility

Functionality

User Experience Impacts the Business

Case StudyImpact of the response time on the conversion rate:

Agenda• Quick Company and Tool Intro• Performance Matters in User Experience• Benefits of Automated Functional Testing• Benefits of Combining Automated Functional and Load

Testing• Live Demonstration

Why Functionalityis important Usability

Ergonomics

Performance

Accessibility

Functionality

Otherwise…

Test AutomationPyramid

Unit

Integration

Functional

Manual

Manual & Integration TestingManual testing• Time-consuming• Error-prone• Cannot „run“ overnight

Integration testing• No black box• Hard to cross technology

borders• Good coding skill required

Benefits of Automated Functional Testing• Quick feedback• Parallel test execution• Scalable

• Codeless functional testing• Increased test coverage

Numbers & FactsTomTom – Case Study:

• Testing time: from 10 days down to 4h• Over 90% of time saved with Ranorex automated tests

Siemens – Case Study:• Due to the use of C#, new test cases can be implemented

more effectively• Only 60% of the time needed with Ranorex compared to

previously used tool

Agenda• Quick Company and Tool Intro• Performance Matters in User Experience• Benefits of Automated Functional Testing• Benefits of Combining Automated Functional and Load Testing

• Live Demonstration

Separated Functional & Performance Testing

• Functional testing ensures functionality… but not when the AUT is under load

• Load/Performance testing ensures a fluent user experience… but does not guarantee a functional UI

Ranorex Customer Experience Survey No Idea

Yes No54% 37%

9%

Do you combine Ranorex test automation with non-functional tests?

In addition to functional test automation you perform…

Other

Load Testing

Database Testing

Web-Service-Testing

Performance Testing

Unit Testing

4%

19%

22%

27%

31%

33%

Functional – Performance Tests

Golden Rule: Automated performance testing should not take place if functional regression tests failed.

Performance tests require that the application or part of the system is:

• Operational: No systematic errors or exceptions

• Representative : Enough data available in the test environment to determine the behavior under load conditions

How Can We Increase Risk Coverage?

100 test cases

40 test casescovering 80% of the risknot necessarily selected for performance testing

We prioritize test cases based on risk. These test cases are validated through :- Automated regression testing- Exploratory testing

Validate Critical Business Case Under Load

This guarantees:• The stability of the crucial use cases under real usage

conditions• High user experience• The identification of potential bottle-necks, even across

technology borders

The Initial Workflow…Automated functional

testing

Exploratory testing

Load testing

Exploratory testing

under load

…

Applying the Ranorex-NeoLoad Integration…

00:00:00 00:00:30 00:01:00 00:01:30

Load

NeoLoad

Application Server

App Error Rate

Virtual Users Load

Automated functional

tests

UserExperience

… You‘ll Find Bugs Earlier!Automated functional

testing

Automated functional

testing under load

Exploratory testing

Exploratory testing under

load

…

Agenda• Quick Company and Tool Intro• Performance Matters in User Experience• Benefits of Automated Functional Testing• Benefits of Combining Automated Functional and Load

Testing• Live Demonstration

Further Resources

Combining Automated Functional and Load Testinghttp://www.ranorex.com/blog/combining-functional-and-load-testing

How to Combine Ranorex and NeoLoad Testshttp://www.ranorex.com/blog/combine-ranorex-and-neoload-tests

Questions?

Henrik Rexed: henrik.rexed@neotys.com

Download NeoLoad Free Edition

Visit our neotys.com

Follow me on Twitter @hrexed

Christian Breitwieser: cbreitwieser@ranorex.com

Download free Ranorex trial

Visit www.ranorex.com

Follow us on Twitter @Ranorex