test automation—it’s a journey, not a project
DESCRIPTION
Organizations implement automated testing in hopes of reducing time and cost while creating higher quality products. They invest in tools, training, and development; identify candidates for automation; and then develop and execute test scripts. Unfortunately, some of these projects are considered failures because the promised savings are not realized. Funding for future automation efforts may be reduced, or automation projects may be cancelled outright. Management support, effective test design and test data management, low maintenance requirements, along with meaningful reports are key to a successful and cost effective automation. Paul Maddison shares how CUMIS has invested more than ten years in test automation—continually growing their investment, reducing costs, and shrinking time-to-market. Paul details their approaches for script design, data management, test candidate selection, and effective reporting to management. Learn approaches that can mitigate challenges and give you a higher return on investment—today and well into the future.TRANSCRIPT
nt Session
Presen d by:
Paul Maddison T
Brought to you by:
340 Corporate Way, Suite Orange Park, FL 32073 888‐2
W7 Concurre4/9/2014 12:45 PM
“Test Automation‐It’s a Journey, Not a Project”
te
he CUMIS Group
300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Paul Maddison The CUMIS Group
Senior quality assurance analyst Paul Maddison has more than ten years of experience in automated testing using a variety of tools. Working with business analysts and testers to identify automation candidates, Paul is continually expanding testing coverage, increasing the return on investment, and reducing regression testing timeframes. He has coordinated the automation team’s development and maintenance of a regression test bed of more than 8,000 scenarios representing about 75 percent of the overall test effort. Recently, Paul designed and developed a self-serve approach to automation execution for use by business analysts and testers allowing the automation team to focus on coding efforts to replace script execution. Contact Paul at [email protected].
1
Test AutomationIt’s a Journey, Not a Project
Paul MaddisonThe CUMIS [email protected]
The CUMIS GroupThe CUMIS Group Limited (CUMIS) partners with credit unions to deliver competitive insurance and financial solutions In doing so it creates financial financial solutions. In doing so, it creates financial security and promotes the growth and success of the credit union system in Canada.
As the leading provider of insurance-related products and services to the Canadian credit union system CUMIS serves approximately 380 credit
2
system, CUMIS serves approximately 380 credit unions, with a total of more than five million members.
2
Getting Started» Resources
–Experienced developers.d f–Aptitude for testing.
–Strong unit testing track record.
» Tool selection–Establish your requirements.–Demo on your software.
k d
3
–Take a test drive.–Report generation & augmentation.–Training availability.
»
Getting Started» Management Involvement
–Visible management support.ff l k h–Current staff may look at project as a threat.
–Establish development milestones.–Celebrate successes.–Communicate, communicate, communicate.
4
3
Return On Investment
5
Test Candidate Selection» Prerequisites
– Reliable test environment.
– Existence of effective manual test cases.
– Availability of subject matter experts.
» Failure Impact– Company credibility.
– Affect on bottom line.
M l T i Eff
6
» Manual Testing Effort– High number of resource intensive test cases.
– Similar test cases with various data combinations.
4
Test Candidate Selection
7
Test Candidate Selection» Effort Savings Formula
XX
8
e.g. 10 minutes * 50 test cases * 3 test cycles =
25 hours of manual testing effort
5
Test Candidate Selection» Insurance Premium Calculations
Coverage Term
9
Script Design» Framework For Reusable Code
– Flexible functions for data population and workflow.S i t i t i d d– Script maintenance is reduced.
– Dynamic environment selection.
» Coding Standards– Common naming conventions.– Internal & external documentation.
» Importing Data
10
p g– Allows for creation using other tools and their features.
6
Data Management» Importance of data driven tests
– Ease of expansion & maintenance.E b dd d f l– Embedded formulas.
– Automation script with multiple data sets.
» Watch Out For Dates– Use of day or year offsets. e.g. Birthdate vs Age
» Formatting– True/False.
11
/– Large numeric.
Automated Script Reporting» Purpose of Reports
– Must be adequate enough to manually reproduce failing test casestest cases.
» Levels of Granularity– Summary Report Contents
» Description of each test scenario and the execution result.
» Number of verifications performed.
» Timeframe required for execution.
Si l S i R t C t t
12
– Single Scenario Report Contents» Data used in test scenario, expected and generated values.
– Execution Log Contents» Identification of failing field, expected and generated value.
7
Summary Report
13
Single Scenario Report
14
8
Execution Log
15
Management Reports» Report Generation
– Source of Metrics» Test execution Summary Reports» Test execution Summary Reports.
» Manual testing candidate evaluations.
– Graphics vs. Numbers» Use of illustrations.
» Additional metrics can be supplied if requested.
» Slice & dice results to generate different views.
– Granularity
16
Granularity» Differentiate between Functional & Regression testing.
» Ensure total automation savings are included.
9
Management Reports
17
Management Reports
18
10
Taking It Further» Test data creation
– Manufacture data files with correct formatting for use in automated tests or for processing in other applicationsautomated tests or for processing in other applications.
» Data Extraction– Extract and save data with specified formatting.
» Environment Smoke Testing– Test connectivity between applications and verify
application functionality before starting a test cycle.
19
» Response Metrics– Compile response metrics for business team review.
Questions?
20