testing framework
TRANSCRIPT
![Page 1: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/1.jpg)
October 15, 2004 – 1
Welcome
IPMA and SolutionsIQ
Professional EventTesting, Testing, 1…2…3…
Improving software quality -- one bug at a time
![Page 2: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/2.jpg)
October 15, 2004 – 2
Agenda
• Building the Test Framework– Jan McCollum, SolutionsIQ
• Break
• Practical Panel Discussion– Cheryl Hainje – AFRS Product Manager, OFM– Dotti Lane – QA Project Manager, OFM– Tim Vessey – POS Project Manager, LCB– Stein Wang – Quality Assurance Lead, SolutionsIQ
• Break
• Testing Templates & Checklists
![Page 3: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/3.jpg)
October 15, 2004 – 3
SolutionsIQ Overview
• SolutionsIQ is a full-spectrum IT services company
• 25 years of technology services and solutions • 400+ consultants• Corporate headquarters (Bellevue, WA)• Professional Services (Bellevue, WA)• Oregon Branch Office (Lake Oswego, OR)• 8+ years of serving the State of WA
– DOC, AOC, LCB, DNR, DOL, LNI, & DSHS
![Page 4: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/4.jpg)
October 15, 2004 – 4
SolutionsIQ Expertise
• Professional Services Division– Consulting and Analytical Solutions
• Project management• Assessments and feasibility studies• Design and architecture roadmaps
– Development and Test Solutions• Full life cycle development projects• Custom application development• EAI, portals, and business intelligence• Quality assurance and testing solutions
![Page 5: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/5.jpg)
October 15, 2004 – 5
Building the Testing Framework
Jan McCollumManager, Quality Assurance and Testing Solutions
![Page 6: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/6.jpg)
October 15, 2004 – 6
Setting Goals
• Knowing WHAT you want is as important as knowing how to get it
– Defining the vision– Defining the timeline– Gaining acceptance and buy in
![Page 7: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/7.jpg)
October 15, 2004 – 7
Defining the Vision
• To define the vision look at what came before
– What went well– What went badly– What now– Where do you want to go
![Page 8: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/8.jpg)
October 15, 2004 – 8
Testing vs. Quality Assurance
• Testing is about finding bugs
• Quality Assurance is about preventing them!
![Page 9: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/9.jpg)
October 15, 2004 – 9
Quality Assurance
• Takes time• Is about the overall effort –
including development• Methodologies can be very formal
![Page 10: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/10.jpg)
October 15, 2004 – 10
QA Applied to Testing
• Quality assurance principals applied to the testing effort will produce higher quality work
![Page 11: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/11.jpg)
October 15, 2004 – 11
Establishing a Timeline
• The 6 month / 1 year / 3 year plan
– Implement processes and strategies that give the best return on investment
![Page 12: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/12.jpg)
October 15, 2004 – 12
Quality Testing Roadmap
• After the goals and objectives are complete, make them real by publishing the quality testing roadmap
![Page 13: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/13.jpg)
October 15, 2004 – 13
Quality Testing Roadmap
• Roadmap should include…
– Test team structure– Communications plans– Test processes– Test procedures
![Page 14: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/14.jpg)
October 15, 2004 – 14
Quality Testing Roadmap
• Test scope• Test dependencies and impacts• Automation transition plan• Test deliverables
![Page 15: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/15.jpg)
October 15, 2004 – 15
Gaining Acceptance and Buy In
• Development• Business management• Project management• IT management• Customer/product support
![Page 16: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/16.jpg)
October 15, 2004 – 16
Making it Happen!
• Organizational structure• Qualified candidates• Roles and responsibilities
![Page 17: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/17.jpg)
October 15, 2004 – 17
Test Planning
• The master test plan: a one-stop shopping guide for your project
– Contents– Contributing documents– Sign-off procedures
![Page 18: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/18.jpg)
October 15, 2004 – 18
Test Planning
• Test matrix and test suites
– Detailed test steps– Pass/Fail results– Tester who performed tests
![Page 19: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/19.jpg)
October 15, 2004 – 19
Test Planning
• Test case design – what is a good test case?
– Accurate – tests what it’s designed to test
– Repeatable, reusable – has a life after this release
– Economical – no unnecessary steps
![Page 20: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/20.jpg)
October 15, 2004 – 20
Test Planning
• Test case design
– Traceable to a requirement– Appropriate for test environment,
testers– Self-standing has enough
information for anyone to run
![Page 21: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/21.jpg)
October 15, 2004 – 21
Test Planning
• Test case design: How to make good test cases better
– Setup, environment, data– Steps, actions and expected results– Use active voice in expected results– System displays this, does that– Simple, conversational language
![Page 22: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/22.jpg)
October 15, 2004 – 22
Test Planning
• Test case design: Why work to improve test cases?
– Productivity – less time to write and maintain cases
– Testability – less time to execute them– Scheduling – better reliability in
estimates
![Page 23: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/23.jpg)
October 15, 2004 – 23
Defect (Bug) Management
• Deciding upon a tool
– Easy of configuration– Ability to add/change fields– Reporting capabilities
• Integrated solution
![Page 24: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/24.jpg)
October 15, 2004 – 24
Defect (Bug) Management
• The bug lifecycle
– Who can create bugs– Who can assign bugs– Who can close bugs
![Page 25: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/25.jpg)
October 15, 2004 – 25
Defect (Bug) Management
• The bug triage meeting– Purpose and who should go
• Reporting– Determining a trend
• Bug metrics– Number of bugs found– Bugs found in production vs. test
cycle
![Page 26: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/26.jpg)
October 15, 2004 – 26
Moving On
• Improving the process: Requirements traceability
– Test cases for each requirement– Requirements matrix– Tracing requirements to defects
![Page 27: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/27.jpg)
October 15, 2004 – 27
Moving On
• Improving the process: Risk-based testing
– You can’t test everything so test what is important
– The risk list and how to use it to drive test strategy
![Page 28: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/28.jpg)
October 15, 2004 – 28
Broadening Your Scope
• Build verification testing– Also called smoke or acceptance tests– Is a subset of the major functional
areas
• Integration testing– Testing the entire system
![Page 29: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/29.jpg)
October 15, 2004 – 29
Broadening Your Scope
• Compatibility testing– How application works with other apps
• Configuration testing– Testing on different configurations
• Setup testing– Testing the installation
• Regression testing– Verify if bug fixes are successful
![Page 30: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/30.jpg)
October 15, 2004 – 30
Broadening Your Scope
• Black box testing
• White box testing
• Grey box testing
![Page 31: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/31.jpg)
October 15, 2004 – 31
Improving Quality
• Testing metrics – measure your success
• Bug tracking metrics– Number found– Number found per component– Daily bug find rate
![Page 32: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/32.jpg)
October 15, 2004 – 32
Improving Quality
• Test case effectiveness– Metric: Test case effectiveness; test
case effectiveness = bugs found in test/total found * 100
• Test coverage– Metric: Test coverage (absolute) =
tests conducted/total tests * 100
![Page 33: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/33.jpg)
October 15, 2004 – 33
Improving Quality
• Test team performance– Metric: Test process effectiveness:
test process effectiveness = bugs fixed/bugs found * 100
– Metric: Planned days vs. actual days in test
![Page 34: Testing Framework](https://reader036.vdocuments.mx/reader036/viewer/2022062512/5549f400b4c9050d488b5637/html5/thumbnails/34.jpg)
October 15, 2004 – 34
Improving Quality
• QA and test involvement early!
• Design reviews– Why testers should attend
• Develop and use checklists
• Project closeout meetings– You should have them