building a test automation strategy for success

© Utopia Solutions

Building a Strategy for Test Automation Success

Lee Barnes, CTO Utopia Solutions

© Utopia Solutions

Agenda

• What is the end goal? • Why do we need a strategy? • What do we need to consider? • What should a strategy look like?

Test Automation Strategy Overview

Interactive Case Study

© Utopia Solutions

SECTION 1 What is Our End Goal?

© Utopia Solutions 4

Test Automation Success Criteria

• Issue detection and recovery • Accurate verification • Unattended execution

Reliable

• Minimum sensitivity to application and test case changes

• Test case definition separate from automation code

Maintainable

• Test coverage expanded efficiently • Automated test cases created by non-

automation resources Scalable

© Utopia Solutions 5

Test Automation Approach

Test Automation Framework

Automated Test Cases Test Results

Test Definition

Test Execution

Non-technical / Test Analysts & Subject Matter Experts

Technical / Automation Engineers

© Utopia Solutions

SECTION 2 Why Do We Need a Strategy?

© Utopia Solutions 8

Why Test Automation Fails

Organizational • Lack of support • Unrealistic expectations • Disjointed org structure • Missing skill sets

Process • Immature testing process • Segregated processes • Lack of test environment / data

control

Technical • Poor tool support for AUT • Under or over design automation

approach

© Utopia Solutions 9

Are We Ready?

© Utopia Solutions

SECTION 3 What Do We Need to Know First?

© Utopia Solutions 11

Test Automation Readiness Assessment

Org

• Structure • Support • Goals and

Expectations • Resources and Skills

Process

• Test Maturity • Test Assets • Process Integration • Test Environment /

Data

Tech • AUT Technology • AUT / Tool

Integration

Assessment Topics

© Utopia Solutions

Where Do We Get the Information?

Assessment Input

Stakeholders

Existing Assets

AUTs

© Utopia Solutions

Ideal Organizational Attributes

Structure

• Centralized testing function

• Testing is a peer of development

• Automation roles are defined

Goals and Expectations

• Automation is an investment

• 100% automation is not likely

• Automation requires specialized skills

• Automation will free up resources to work on high-value tasks

Support

• Executive management has visibility to automation

• Automation has a dedicated budget

Resources and Skills

• Resources are dedicated to automation

• Resources are skilled in automation as a process AND technology

© Utopia Solutions

Ideal Process Attributes

Test Process

• Test objectives are clear and measurable

• Test cases are defined and documented

• Expected results are predictable

SDLC Integration

• AUT in a frequent release state with visibility to future roadmap

• Visibility to system changes as early as possible in release cycle

Test Environment

• Test environment dedicated to automation

• Test data controlled

© Utopia Solutions

Ideal Technology Attributes

AUT Technology

• Multiple AUTs built on common technology and/or development framework

• Consistent use of UI objects

• Consistent navigation • Consistent presentation

of user feedback (warnings, errors, etc.)

Tool / AUT Integration

• All UI objects supported by tool

• Any alternative interfaces (web services, API, DB, etc.) supported by tool

• All target end user environments supported by tool

© Utopia Solutions

SECTION 4 Common Assessment Findings

© Utopia Solutions 17

Organizational

Distributed Testing /

Automation Efforts

App App

App

App • Uneven automation

resource demand • Inconsistent approach • Duplicated efforts • Restricted skill growth

and knowledge sharing

Leads to…

© Utopia Solutions 18

Organizational

AUT A

AUT B AUT C

AUT D Centralized

Test Automation Team

Common Resources, Processes & Technology

• Flatten demand curve • Accelerate skills

development • Leverage reusable

knowledge and technology

Benefits

Centralize Automation Activities

© Utopia Solutions 19

Process

Analyze Design Build

Test Deploy

No ability to identify impact of app changes on automated test suite

Unexpected / Unpredictable Maintenance

© Utopia Solutions 20

Process

Analyze Design Build Test Deploy

Capture potential impacts as early as possible to enable proactive maintenance

Capture App Changes

Analyze Impact Maintain Execute

Automation Activities

Application SDLC

© Utopia Solutions 21

Environment

Shared Test Environment

Uncontrolled Test Data

Integration

Functional

Performance

Integration Test

• Schedule conflicts • Unreliable execution • Unnecessary “hardening”

of the test suite

Leads to… Automation

© Utopia Solutions 22

Environment

Shared Test Environment

Controlled Test Data

Integration

Functional

Performance

Integration Test

Dedicated Environment for Automation

Test Data Controls

Dedicated Environment

Controlled Test Data

Automation

• Restore to known state • Partition data related

to specific test activities

• Create required data each execution

© Utopia Solutions

SECTION 5 What Does a Strategy Look Like?

© Utopia Solutions 24

Mobile Test Strategy Topics

Test Automation

Strategy

Process & Organization

Environment

Technical Resources

Scope & Roadmap

© Utopia Solutions 25

Automation Strategy – Process and Organization

• Political and financial support • Realistic expectations • Centralized automation function

Key Organizational Components

• Test objectives and coverage goals • Integration with SDLC → proactive

maintenance

Key Process Components

© Utopia Solutions 26

Automation Strategy – Environment

• Dedicated environment for execution • Controlled test data • Controlled code migration

Key Environment Components

© Utopia Solutions 27

Automation Strategy – Resources

• Roles • Skills • Time

Key People Components

• AUT test environment(s) • Test suite development • Test suite execution

Key Hardware Components

• Test tool • Supporting utilities

Key Software Components

© Utopia Solutions 28

Automation Strategy – Scope

• Test data setup • AUT / environment preparation • Complex / tedious results verification

Test Support Tasks

• Applications built on common technology • Applications with frequent releases & friendly UI’s

Application Selection

• Frequently tested • High test case to UI ratio • Predictable results • Easy to automate

Test Case Selection

© Utopia Solutions 29

Automation Strategy – Roadmap

• Begin with required organizational and process changes • Overlap with base framework and minimal coverage • Enhance and add test coverage over small iterations

Create Implementation Timeline

Phase n Phase 1

Continuous Process & Organizational Improvements

Base Framework

Initial Coverage

AUT 1

Framework Enhancements

Expand Coverage

Initial Coverage

Phase 2

AUT 2

Expand Coverage

Expand Coverage

Initial Coverage

Phase 3

AUT 3

© Utopia Solutions

SECTION 6 Case Study

© Utopia Solutions 31

Key Assessment Findings

Organization • All enterprise testing activities

currently performed centrally • Management believes automation

is a tool implementation

Process • Manual tests are well structured,

but not maintained consistently • Rely on SME’s to execute tests

Environment • Manual regression test performed

in a isolated environment • Test data refreshed in all

environments once per quarter

Technology • All AUTs have browser based UI • Developed on disparate

technologies • Test tool supports all AUT

components

Resources • Test organization consists of non-

technical SME’s • System performance (mobile) • Device performance

© Utopia Solutions 32

Key Summary Points

• DO NOT automate without a clear vision and plan • Understand your readiness from these perspectives

• Organization • Process & Environment • Technical • Resources

• Implement in small phases

© Utopia Solutions 33

Questions…

…. and Answers!

Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: lee.barnes@utopiasolutions.com Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com