test automation lessons from websphere application server

© 2010 IBM Corporation

Robbie Minshall, [email protected]

WebSphere Test Story

IBM Software Group


Agenda Test Overview Meaningful Regression Usage of Cloud Resources

2

IBM Software Group


Test OverviewOverview of the WebSphere Test Organization Scope

3

IBM Software Group


WAS End to End Testing OverviewBuilding Blocks to a Quality Release

Unit BVT FVT / CT SVT CTS Performance Usability GVT / TVT Service Stream

• Build Verification Test (Rapid Regression) - 20,000+ tests across all releases

• Function Verification Test (Daily Regression) - 1.7+ million tests

• System Verification Test - 1100+ complex customer scenarios tested

• Compatibility Test Suite - 62,000+ Java certification tests

• Performance Test - Over 15 performance benchmarks (including end to end workloads like DayTrader and SpecJEnterprise) executed in various configurations and scenarios

• Globalization Verification Test - Complete GVT on 9 distinct languages

• Post GA Testing• JDK and OS certification testing• Service Stream Testing including FVT regression and long runs and stress testing for each fixpack including performance analysis• APAR test validation• CTS regression execution

4

IBM Software Group


Significant scope Significant resources

• Between 200 + engineers involved, 55+ dedicated

• 3000 + cores , 500+ z/OS systems on 10+ LPARs, 45+ iSeries LPARs Every day

• 1.7 million + tests executed every day in a meaningful regression

• 16 + hours of automated security variations

• 8+ database variations Throughout each iteration

• Performance regression patrol

• Compliance testing

• Cross platform on over 25+ OS variations

• Stack and personsa Release

• Compliance

5

IBM Software Group


Testing is NOT an Ad-Hoc event...It is a Science - a Repeatable Closed-Loop Process

Test Inputs:•Use Cases•Non-functional attributes•Design documents•Schedules•Customer/Business Partner input•Problem Analysis•etc.

Test Outputs:•Comprehensive Test Plan•Test Scenarios•Test Cases•Test Configurations•Status

Test Execution•Assign Test cases to test engineers•Assign Test scenarios to configs•Execute tests•Write defects•Debug/provide data•Analyze results•Track Status

Post Analysis :•Analyze the test results•Determine what worked/didn't work•Work with Support to find escapes•Work with Customers to see how closely our use cases match theirs

6

IBM Software Group


Agile Transformation

7

Traditional Waterfall

Agile

VS.

IBM Software Group


Despite Dilbert … We’ve Seen Benefits!

8

IBM Software Group


Transition From Waterfall

Cycle time reduction from 7 iterations to 1 iteration

60% reduction of resources for functional verification

SVT executed each Iteration

WaterfallCycle time = 7 iterations

Development Iterations

Code

Comp FVT

Test Cycle

Auto FVT

AgileCycle time = 1 iteration

Iteration 1

Code

Auto Dev

Iteration 2

Code

Auto Dev

Iteration n

Code

Auto Dev

Continuous Automated Regression

SVT Persona

SVT / Persona

Regression detection down from 3 months to 1 day

Regression execution time 6 weeks -> 1 day

80% reduction in downstream regression

9

IBM Software Group

© 2010 IBM Corporation10

Leveraging the Cloud

Find Hardware

Setup Automation Deploy Validate Work

Deploy Pattern Work

Fast deployment in minutes

Spend less time administering, more time developing and testing new solutions

Take repeatable tasks and automate

IBM Software Group

© 2010 IBM Corporation11

A concise descriptive model of a company, what it wishes to accomplish, and why.

A composite archetype based on behavioral and descriptive data gathered from many actual companies that share related usage patterns.

Our customer personas are intended to provide WebSphere development and testers with a consumable source of customer information that can serve as a context for writing more customer-oriented test scenarios.

Better understanding of customers = more effective design and testing

WAS System Test Customer Personas

IBM Software Group


Summary Quality is a key focus area

• Organization is accountable for quality

• Test organizations are responsible for driving repeatable performance and quality disciplines throughout the organization.

Complexity is now the 'norm’• Test efforts need to factor this customer environment complexity

• Our offerings need to bring new levels of simplicity Evolve

• APAR Analysis

• Customer intimacy and engagements Product Integration

• Its about solutions and tooling

12

IBM Software Group


Meaningful functional regressions

Driving value and achieving quality through Meaningful Regression

IBM Software Group


The cost of regressions Cost goes up exponentially with time before detection Communication overhead between users/test and developers

• Example : Over 50% time between detection and fix spent ‘communicating’ Tangled dependencies

• Building function on top of bugs results in new regressions and slow progress Can not start enterprise testing w/o functional stability

• Results in long enterprise testing phase slowing delivery

• Enterprise issues not discovered till too late to refactor or redesign

• Often reduces enterprise testing to verification rather than limit based

Production Costs are huge

14

IBM Software Group


The need for speed

Rapid development requires repeatable quality. “I don’t have time to write automated repeatable tests we are on a tight

development schedule here” Development schedules are irrelevant. Delivery and quality counts. How can you go fast and have high quality… you can ONLY go fast if you

have consistent high quality. You must :

#̶ Be able to refactor

#̶ Detect regressions immediately

Why : #̶ There is no time to recover, to build fault on top of faults or to release/distribute something of

unknown quality.

Repeatable quality must be the foundation of how you do business.

15

IBM Software Group


Traditional Quality Assurance

16

What : Develop codeUnit TestLeverages : DesktopsCosts : $$

Dev

What : Develop codeUnit TestLeverages : Loose LabCosts : $$

FunctionTest

What : IntegrationScaleLeverages : Prod Lab 2 Costs : $$$

Pre-Prod

What : BenchmarkIdentifyLeverages : Isolated LabCosts : $$$

Perf

What : ProductionMonitorLeverages : Prod LabCosts : $$$

Prod

What : DiagnoseFixLeverages : Prod Lab (2)Costs : $$$$

Service

IBM Software Group


Pressures on quality

17

Dev

Pre-Prod Perf ProdService

Dev

Dev

Function Test

Function Test

Function Test

Dev

Function Test

Pre-Prod

Perf

ProdService

Service

IBM Software Group


How to respond How can you go fast and have high quality ?

• You can only go fast if you always have high quality Can Agile methodologies help me ?

• Agile development is dependent upon agile operations/infrastructure How can I scale performance engineering ?

• Move selected tasks into development

• Availability of consistent configured environments

• Low cost for infrastructure

1805/03/23

IBM Software Group


Unit Tests Generally accepted by development organizations as well as test

organizations a good programming practices

Provides protection against component regressions Based on testing smallest testable unit of an application

Should be run as an automated part of your build. When there is a failure. Stop and fix. Then move forward.

19

IBM Software Group


UnitTest Limitations Legacy Code

• Often difficult to break into testable units Units dependencies require expensive scaffolding

• Other units/components

• External resources like databases

• Middleware environment

UnitTest is not : • System Integration Test

• Replacement for integration and enterprise testing

UnitTest is great. However, it tests the parts not the sum.

20

IBM Software Group


Integration Acceptance Tests WebSphere calls these Functional Acceptance Tests but Integration

Acceptance Tests ( IAT ) is more descriptive

An IAT is a fully automated system integration test.

Include micro benchmarks as performance regression tests.

Forms the cornerstone of meaningful functional regression

Same concepts as UnitTests but focused on systems integrity

21

IBM Software Group


Continuous Test Test Continuously using automation

In response of every build invoke regression tests

Daily Meaningful Functional Regression Continuous Long Runs, stress and applications

Establish a culture and policy of functional stability̶# Similar to historical issues with compilation/build stability

22

IBM Software Group


Invest in quality Consistent

• Must be repeatable Portable

• Test cases should not be dependent upon a complicated or non distributable test environments

• Developers, testers, stakeholders should be able to easily execute test assets Robust and flexible

• Detection over hardcoding (example: detect ports don’t assume)

• Graceful exits if topology not appropriate

23

IBM Software Group


Roadmap Standardize on Integration Environments

• Defined

• Automated

• Accessible Standard Integration Environment should be for 90%

• Add limited set of extended topologies for prioritized edge cases Standardize on format for unit and integration tests

• Results in simplified automation Implement an end to end integration test

• Use this as initial regression baseline and example Test Continuously Implement integration tests for legacy function based upon your

investment areas and your bugs heatmaps• Use 80/20 rule

24

IBM Software Group


A living regression The true power of a functional regression is realized when it is a part of

your development processes

Myth: Developers do not want to write or execute test cases• Fundamentally this is a problem of accessibility.

• Motivational and philosophical opposition have emerged as a result of poor accessibility

Evolve your regression• All new function must have integration tests

• Culturally may be difficult and may need to start with unit then build up

25

IBM Software Group


Executable Bug Specifications Much time is spent describing bugs imprecisely in English Often not reproducible

• “It works on my machine” …

Define the issue with a unit and/or integration test• Creates an executable specification

• Ensures no future regressions

26

IBM Software Group


Cost for Integration and Continuous Test Hardware and lab management costs

• Consolidate, onDemand access, provisioning Reduced development time

• Standardize topologies, provide common environments

• Accessibility of environments

• Measure and address perception issues Infrastructure consistency

• Repeatability of deployment across platforms can be difficult

• Virtualization or OS Provisioning can help significantly

It is worth it

27

IBM Software Group


Measure Why

• Understand your project and organizations costs

• Assess if change is effective Measurements should not dictate decisions but should help guide and

support them. Let the data speak.

The power of data in influencing change

Development organizations are based on projected revenue of projects. Understand and leverage downstream quality costs to effect change.

28

IBM Software Group


Example Metrics Organizational

• Cycle time ( concept to delivery )

• Production rate ( i.e story points per month )

• Cost of post production bugs vrs pre production bugs

• Days of functional stability ( no regressions )

• Hardware expenses

• Hardware efficiency Test

• Execution time: Time it takes for organization to perform a full regression

• Continuous Test: Time between fix integration and automated regression

• Wetness: Time between regression and detection

• Infrastructure Reliability : # failures due to infrastructure not function

29

IBM Software Group


Summary Apply unit test principals to integration test Automate.

• Invest in repeatability Standardize

• For consistency and simplicity Strive for a full/meaningful regression Test Continuously.

• Find regressions early Measure

• assess value and justify costs Virtualization

• Repeatability, Availability and cost reduction Evolve

• Executable specifications, new integration acceptance tests for all new userStories

30

IBM Software Group


Cloud

Cloud Adoption to support Agile and Functional Regression Objectives

IBM Software Group


Strategic Motivations – Get Agile

32

IBM Software Group


Tactical Motivations

33

IBM Software Group


Agile Transistion

Dev & Integration Test

System

Automated Daily Regression

Iteration Iteration Iteration FVT XP System

Development Teams

34

IBM Software Group


Functional Acceptance Tests Automation Package

• Zip. Ant. Wsadmin. Standardized and Extended Topologies

• Simplify broader automation framework Simple & Portable

• Non framework specific Bootstrapping

• Concise standardized bootstrapping

• Topology Detection Persona Applications

• Dynamic DB

• Application validation

35

IBM Software Group


Changing objectives Performance

• Benchmarking

• Automated regression detection

• Micro-benchmarking and tooling for developers System Test

• Complex WAS topologies and scale

• Limit Testing

• Solution and stack Example

• Passive goal : • Pass 7 day long run

• Limit based goals : • Establish 24 hour stress threshold • Establish long run limit at established stress threshold

36

IBM Software Group


FIT

Typical Iteration

TimeLine

Release

App Owner

I d e n t i f y A p p l i c a t i o n

STRESS

APPFAT APP

FATStory

Iteration

APPFAT APP

FATStory

Pla

n

Inte

grat

ion STRESS

P l a n I n t e g r a t i o n

Extendable Stress Environment

Iteration

....

....

....

PlanIntegration

Personas / Customers

Custom PersonaEnvironment Continuous Varied Run ....

Dem

o & FeedbackLongevity

IBM Software Group


Quality in development

38

IBM Software Group


Cloud Requirements

39

IBM Software Group


Primary takeaways Cloud

• Delivery mechanism for content Be wary of image libraries

• Customized images are hard to manage/service

• Image sprawl Flexibility

• Maintain flexibility about how software bundles are delivered

• Standards for multiple Cloud Providers ( performance, cost, quality )

40

IBM Software Group


Governance Virtualization does not change behavior; it enables change

• Image sprawl

• Usage abuse

Invest in governance rather than restrictive policy Self service with assurance and control

41

IBM Software Group


Land Of Opportunity

Startup • 3 day training for build, 4 week for pre-integration

• Hardware constrained Developer Environments

• 75% PMRs avoidable

• 6 weeks old. 20% have pre-integration setup Automation Complexity

• 50% End user, 18% Service installs fail 6 week functional regression

42

IBM Software Group


Return on Investment

43

IBM Software Group


ROI: Labor Savings

44

SimplificationSavings

IBM Software Group


Labor : Setup an Environment

Find Hardware

Setup Automation

Deploy Validate Work

Deploy Pattern

Work

45

IBM Software Group


Drastically reduce set up and configuration time– New environments in minutes!

Reduce risk by codifying infrastructure– Freeze-dry best practices for

repeated, consistent deployments

Security throughout the entire lifecycle

Simplify maintenance and management– Flexibly manage and update the components of your patterns– Ensure consistency in versions across dev, test, production

Spend less time administering, more time developing new solutions

What does this mean ?

46

IBM Software Group


Usage Scenarios Organizational Access OnDemand Topologies Self contained Env Pre-Integration / Dev

Automated Regression Migrate Patterns Elastic Capacity Automated Daily Regression

Dev & TestAutomation

Performance and System Test

AutomationWebUI CLI

47

IBM Software Group


Returns on Investment

48

IBM Software Group


ROI: Labor Savings

49

SimplificationSavings

IBM Software Group


Labor : Setup an Environment

Find Hardware

Setup Automation

Deploy Validate Work

Deploy Pattern

Work

50

IBM Software Group


Results of investing in quality through development Cycle time from 7 iterations to 1 iteration Regression detection down from 3 months to 1 day Regression execution time 6 weeks -> 1 day SVT executed each Iteration 60% reduction of resources for functional verification 80% reduction in downstream regression

"Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. Actual performance in a user's environment may vary."

51

IBM Software Group


Labor Savings

5 year return on investment 10,455,224

IBM Software Group


WebSphere deploys private cloud driving Agile Adoption and gain efficiencies throughout delivery

Client Pains Enable rapid access to complete testing

topologies to improve quality and decrease costs

Drive down costs and increase utilization of infrastructure

Real Results 1.7 M in direct savings, 2.1 M/year in enabled efficiency. Reduced topology install time from 3 Hours to 20 Minutes.

Increased hardware utilization by 54% . Increased System Admin efficiency by 16X compared with bare metal.

Interact