test automation: a roadmap for sucesss
DESCRIPTION
Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.TRANSCRIPT
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 1
SQS Group Limited
Scott Armstrong
Belfast, February 2010
Test Automation
A Roadmap For Success
Agenda
� Steps for Success
� Automation Tool Selection
� Test Selection and Prioritisation
� Implementing the Strategy
� SQS Case Studies
� Questions
2© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 2
Steps for Success
Approaching Test AutomationPlanning the Framework
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Approaching Test AutomationSteps for Success
AUTOMATION PROJECT
DESIGN
Build maintainable tests
Target and prioritise test areas
TOOL
Select with care
Pilot tool
Manage Implementation
TEST MANAGEMENT
Defect Management
Config Management
Information and Reporting
Requirements
TEAM
Code reuse and collaboration
Partnership between test and dev
PLAN
Treat as a Development Project
Manage Expectations
Commitment at all levels
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 3
Planning the FrameworkSteps for Success
� What are the Management Information Requirements
� Contents
� Format
� What are the Execution Requirements?
� On-demand
� Automated Nightly Builds
� What technologies are we testing?
� Embedded
� Operating System
� What resources are available?
� Hardware
� Technical Expertise
� Are there cultural or organisational considerations?
� Outsourced IT Management – no admin access
� Current partnership with tool vendor
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Automation Tool Selection
Selection ProcessIdentify RequirementsBuy, Open Source or Build
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 4
Selection ProcessAutomation Tool Selection
Define Problem
Consider automation
as a solution
Make business
case
Define required features
Shortlist
Evaluate
Demo
Trial
Decide
Constraints
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Identify RequirementsAutomation Tool Selection
� Start by evaluating your requirements, not by investigating the tool market.
� Look at the current problems – what has prompted the search for an automated solution:
� Manual testing problems (e.g. too time consuming, error prone);
� No time for regression testing when small changes are made to the software;
� Set-up of test data or test cases is error prone;
� Inadequate test documentation;
� Don’t know how much of the software has been tested;
� Testing is ineffective.
� Not all of these problems are best solved by automation.
� Rank your problems by importance or current cost to the organisation.
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 5
Buy, Open Source or BuildAutomation Tool Selection
• Quickly provides a high level of features
• Well supported in terms of documentation, help and training
• Often an attractive project to be involved with
• Added cost of license fees
Commercial Tool
• No license fee cost
• Limited to support from community forums
• Defects in tool may not be addressed
• Tool may lack a clear roadmap; may not be future-proof
Open Source Tool
• No license fee cost
• Development and maintenance can prove expensive
• Designed to meet project requirements
• Lacks full range of features
Build Bespoke Tool
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Test Selection and Prioritisation
Test SelectionTest Prioritisation
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 6
Test SelectionTest Selection and Prioritisation
� Some tests cannot be automated.
� Some tests are not worth automating.
� May take more time to automate tests than to run them manually
� You may not have to automate everything to get significant benefits
� 10% of the tests may be run 90% of the time.
� Some types of tests are amenable to automation, some not:
� Functional tests are often the most straightforward
� Performance tests are notoriously difficult to carry out manually. These are a good candidate for automation.
� Non-functional tests can cover maintainability, portablility, testability, usability, etc, and can be inappropriate for automation.
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Test PrioritisationTest Selection and Prioritisation
� Do not automate too much too soon.
� Best practices take time to develop.
� Go for ‘quick wins’ – the areas where automation will have the largest impact most quickly.
� Prioritisation factors:
� A set of breadth tests (sample each system area);
� Tests for the most important functions;
� Tests that are easiest to automate;
� Tests that will give the quickest payback;
� Tests that are run the most often.
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 7
Implementing the Strategy
Automation Team FitMaintenance Model
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Automation Team FitImplementing the Strategy
Automation Team
Specialist Group/Tool
Support
Subject Matter Expert
Management Buy-in
Development Team
Test Team
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 8
Maintenance ModelsImplementing the Strategy
• Client representative with technical background is trained during automation project
• Maintenance of current scripts can be supported independently for future product releases
• Future new tests can be automated in-house and added to the regression pack
Knowledge Transfer
• Client does not have to be involved in the details of automation development, execution or maintenance
• The necessary technical skills do not need to exist in-house
• Assured best practice automation and quality
Managed Service
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
SQS Case Studies
Lloyds TSB Wholesale BankingLocal Utility Company
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 9
Lloyds TSB Wholesale BankingSQS Case Studies
Manual Testing Challenges
• 600+ Tests and growing
• 200 man days per release
• Concerns over coverage
• Regression pack growth impacted by limited resources
Our Engagement
• Innovative test automation solution implementation
• Highly maintainable and extensible solution
• Successful solution handover to client test team
Customer Success Return on Investment
• Increased test execution efficiency by 85%
• Reduction in deployed defects >50%
• Enabled daily smoke test execution
• Overall ROI of £1.9m over 6 years
Lloyds TSB
“Without
automation, we
would not have
gone live”
“The automated
regression suite
made my job
more interesting”
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
Local Utility CompanySQS Case Studies
Manual Testing Challenges
• 100 test scripts with multiple complex data-driven iterations
• 120 man days regression testing per year
• Low test coverage – limited to ‘spot checks’
• Testing of new functionality impacted by large regression packs
Our Engagement
• Collaborative design of framework specific to client requirements
• Full knowledge transfer to existing test team for ongoing maintenance and development
• Modular framework allows new tests to be built with minimal effort
Customer Success
• SQS are now engaged on further automation projects with the parent company
• The existing test team continue to expand and maintain the framework to increase their ROI
Return on Investment
• Saving of 96 man days of execution per year
• Data-driven scripts increased test coverage by a factor of approximately 200
• Automation solution was ported to a further 2 business units, multiplying the ROI
Local Utility
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010
2008-01-20
© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 10
Thank you for your attentionThank you for your attention
SQS Group Limited
The Concourse | Northern Ireland Science Park
Queen‘s Road | Queen‘s Island | Belfast BT3 9DT
Phone: +44 (0) 2890 022 660
E-Mail: [email protected]
Internet: www.sqs-uk.com
© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010