relieveing the testing bottle neck - webinar
TRANSCRIPT
![Page 1: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/1.jpg)
September 25, 2015
Introduction to QASymphony for
[INSERT COMPANY NAME]
Relieving the Testing Bottle Neck in Your
Projects
![Page 2: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/2.jpg)
AGENDA
• How Testing Becomes a Bottleneck• Removing the Bottleneck with a New Process• Increasing Manual Testing Efficiency• Improving Automated Testing Coverage &
Reliability• Build Testability into the Application• Managing an Testing Strategy• Measuring a Development Team on Quality• Q&A (based on interest)
![Page 3: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/3.jpg)
September 25, 2015
Introduction to QASymphony for
[INSERT COMPANY NAME]
Why does Testing become a Bottleneck?
![Page 4: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/4.jpg)
Past development cycles often modeled the Rational Unified Process:
Source: http://www.psa-software.com/_img/_knowledge_center/rup.jpg
WHERE WE CAME FROM
![Page 5: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/5.jpg)
If testers cannot test efficiently with predictable estimates of effort, testing can become the bottleneck for development (less features deployed):
TESTING CAN BECOME A BOTTLENECK
Design Code Test Deploy
Features that couldn’t be tested, so not deployed
![Page 6: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/6.jpg)
Given a fixed level of depth in the testing , we can test and deploy more features but it will be less coverage
WHAT IF WE TEST LESS?
Design Code Test Deploy
Reduced depth of testing on features
![Page 7: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/7.jpg)
If given the choice, we would have moved testing earlier in the cycle, but we were constrained by many environmental factors:
WHY DOES TESTING NEED TO BE LAST?
Environment Creation Code Merges On-Premise
Prevalence
Desktop Focus
Lack of Collaboration
Off-Shore Development
![Page 8: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/8.jpg)
Many of our prior limitations have been replaced, based on macro trends around technology and industry:
WE CAN MOVE THE TESTING UP FRONT
• Containers have simplified the process dramaticallyEnvironment Creation
• Git has replaced Subversion as the industry standardCode Merges
• Cloud adoption is at an all time high, increased securityOn-Premise Prevalence
• Prevalence of Web, Mobile, Internet of thingsDesktop Focus
• Increase in teamwork, chat and collaboration technologyLack of Collaboration
• Shifts towards rural sourcing, onshoring of laborOff-Shore Development
![Page 9: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/9.jpg)
Now that we have freed ourselves of past limitations, the process can be shifted to one that aligns more with our needs:
Traditional Approach
Test-First Approach
HOW THE PROCESS CAN LOOK NOW
Design Requirements Code Test Deploy
Design (Automated) Test Code Refactor Deploy
![Page 10: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/10.jpg)
Creating tests up front means code can be deployed almost right upon completion, removing the testing bottleneck:
HOW THE FUNNEL LOOKS NOW?
CreateTests
Code Deploy
![Page 11: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/11.jpg)
Moving testing to the front of the process is desirable, but many realities of software teams restrict them from doing so: • Lack of automation coverage for legacy features• Lack of testability (API’s, UI frameworks) in legacy
applications• Lack of automation talent within the organization• Finite project length (i.e. package software
implementation)• Regulatory compliance (need to document requirements)
For these organizations – we will also cover ways to make a traditional requirements driven approach more efficient
FLIPPING THE FUNNEL MAY NOT BE POSSIBLE
![Page 12: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/12.jpg)
September 25, 2015
Introduction to QASymphony for
[INSERT COMPANY NAME]
For Manual Testers
![Page 13: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/13.jpg)
Manual test execution can be optimized by implementing exploratory testing. A 2007 controlled study found that:
– Testing with test cases vs. exploratory testing take almost 7 times longer, due to the amount of time needed to write the tests and report on them – TEST FASTER
– Testing with test cases vs. exploratory testing doesn’t find any more defects, and does not miss many (if any) critical or severe defects in comparison to test case testing – TEST BETTER
– Testing with test cases causes more false defect reports vs. exploratory testing – TEST SMARTER
Study link: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.3696&rep=rep1&type=pdf
HOW TO OPTIMIZE MANUAL TESTING
![Page 14: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/14.jpg)
1. Parallel test planning, test design, and test execution
2. Specific yet flexible
3. Aligned towards investigation of potential opportunities
4. Values depth and attention to detail during testing
5. Fosters knowledge sharing and awareness
WHAT IS EXPLORATORY TESTING?
![Page 15: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/15.jpg)
Unlike traditional testing techniques, planning, design, and execution happen concurrently, allowing efficiencies of time as well as flexibility in approach
PARALLEL PLANNING, DESIGN & EXECUTION
Plan Design Execute Report
Plan
DesignExecute
Report
![Page 16: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/16.jpg)
Exploratory testing provides a specific lens through which to perform testing – whether that be a user persona, functionality, criteria (i.e. localization), etc.
However, it allows testers to use the tool as an end user would, not necessarily as the product owner envisioned it
SPECIFIC, YET FLEXIBLE
Manual Scripted Testing
I tested the application as the script prescribed
Exploratory Testing
I tested the application as the end user would
![Page 17: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/17.jpg)
Exploratory testing rewards testers who identify unknown areas of “opportunity” within the application, as they are essential in maintaining a backlog of future test charters
INVESTIGATING OPPORTUNITIES
Manual Scripted Testing Exploratory Testing
![Page 18: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/18.jpg)
Exploratory testing relies on knowledge sharing to reach full potential – developing testers who understand the impact of more areas of the application allows them to identify more areas of risk and opportunity
KNOWLEDGE SHARING
Plan
DesignExecute
Report Transfer Learning
Example Questions to Ask
• Have you seen this before?
• What am I not considering?
• Why would someone do this?
• How would you have tested this?
![Page 19: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/19.jpg)
September 25, 2015
Introduction to QASymphony for
[INSERT COMPANY NAME]
For Automated Testers
![Page 20: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/20.jpg)
Moving towards TDD will also demand a more complete automated testing strategy focused on more than just UI testing:
BUILDING A COMPLETE TESTING STRATEGY
UI Tests
Integration TestsU
nit Tests
UI TestsIntegration
Tests
Unit Tests
![Page 21: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/21.jpg)
UI automation should be only be used for key features and functionality, given the following limitations:
• SPEED – tests take much longer to run when waiting for pages to load, objects to interact, etc.
• RELIABILITY – tests break frequently for reasons besides failing functionality (like renamed or moved objects)
• MAINTENANCE – tests require frequent updating as UI elements are changed, requiring more maintenance
• COST – many UI automation tools are expensive and based on perpetual licensing
WHY UI AUTOMATION IS LIMITING
![Page 22: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/22.jpg)
The ideal testing pyramid would look like the following:
THE IDEAL RATIOS
UI TestsIntegration
Tests
Unit Tests 70% of coverage, effort, maintenance
20% of coverage, effort, maintenance
10% of coverage, effort, maintenance
![Page 23: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/23.jpg)
Focus on the missing box (es) in the Agile Testing Matrix - typically the performance and load tests:
Source: http://image.slidesharecdn.com/agiletestingandtheroleoftheagiletester2008-09-26rev4-090615163750-phpapp01/95/agile-testing-the-role-of-the-agile-tester-40-728.jpg?cb=1245084455
ADDING OTHER MISSING TESTS
![Page 24: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/24.jpg)
September 25, 2015
Introduction to QASymphony for
[INSERT COMPANY NAME]
For Developers
![Page 25: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/25.jpg)
Moving to TDD process moves testing activity from solely UI based to focusing more testing at the services/API layer:
BUILDING TESTABLE SOFTWARE
Need for Testability
![Page 26: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/26.jpg)
Testing in the Unit Layer will require more mock objects to “unhinge” features that are being tested, removing them from dependencies:
Source: http://zeroturnaround.com/wp-content/uploads/2015/12/PUZZLE-1-min.png
BUILDING TESTABLE SOFTWARE
![Page 27: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/27.jpg)
Many teams struggle to pick an automation framework, but the reality is most frameworks are similar and differentiate in a few key areas:
PICKING AN AUTOMATION FRAMEWORK
•Does it support dev centric languages (i.e. Java) or easier to use languages (i.e. Ruby)
•Does it support the concept of features, or stories?
•Does it have features to support large enterprises, as well as small teams?
• Is it more focused on BDD, TDD, or ATDD?
Dev Style Scalability
LanguageWorkflow
![Page 28: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/28.jpg)
September 25, 2015
Introduction to QASymphony for
[INSERT COMPANY NAME]
For Test Managers
![Page 29: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/29.jpg)
There is a myth that EVERY test should be automated immediately for a team to be as agile as possible
It is impossible that creating an automated test would the most efficient approach in all scenarios
Some examples:• Tests being used one time• Tests on difficult parts of an application• Tests for parts of the system that change frequently
THE AUTOMATE EVERYTHING MYTH
![Page 30: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/30.jpg)
A more balanced approach is to estimate the total cost of creating AND maintaining a test over a course of its life vs the cost of manual testing:
AN ROI BASED APPROACH
Automation Manual
Script Creation Cost $500 $200
Script Maintenance Cost $200/year $50/year
Script Execution Cost $120/year $480/year
Cost after Year 1 $820 $730
Cost after Year 2 $1140 $1260
Cost after Year 3 $1460 $1790
Manual
Even
Automate
![Page 31: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/31.jpg)
For teams that are looking to build automation coverage on legacy applications, another approach might be to automate based on time saved
A SAVINGS BASED APPROACH
Test Case A Test Case B Test Case C
Manual Execution $500/year $300/year $400/year
Manual Maint. $200/year $100/year $200/year
Auto. Creation $300 $150 $480
Automation Maint. $100/year $50/year $100/year
Year 1 Savings $400 $250 $120
![Page 32: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/32.jpg)
September 25, 2015
Introduction to QASymphony for
[INSERT COMPANY NAME]
For Dev Managers
![Page 33: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/33.jpg)
Agile promotes delivering software quickly, so development managers often incentivize developers based on features, story points, or lines of code delivered
However, this approach can often incentivize bad behavior – pushing incomplete or unstable code over to testers, who are then responsible to identify the issues developers missed
Development managers should incentivize their teams for building good code quickly, which is understandable and easy to test and debug
MANAGING ENGINEERS TO QUALITY METRICS
![Page 34: Relieveing the Testing Bottle Neck - Webinar](https://reader030.vdocuments.mx/reader030/viewer/2022032711/588299291a28ab92618b4917/html5/thumbnails/34.jpg)
Coupling Between Objects (CBO) – A measure of how closely or loosely coupled methods are. 1-4 is good, higher than that may mean risk in trying to refactor or resolve issues
Comment Ratio – Ratio of comments to total lines of code, indicating how much the developer has documented the code for future debugging or refactoring
Associated Critical Defects Ratio – Ratio of critical defects opened on specific features divided by the effort (story points) for that feature. Often normalized for feature complexity
SOME GOOD QUALITY METRICS TO TRACK