test plan simplicity

ConfidentialPA12013-11-121 ConfidentialPA12013-11-121

Test PlansSimplicity

ConfidentialPA12013-11-122

Introduction• In this presentation I will use James Bach’s

definition of a test plan [1]• Test Plan: the set of ideas that guide a test project

• Test Strategy: the set of ideas that guide test design

• Test Logistics: the set of ideas that guide the application of resources to fulfill a test strategy

• Every artifact created must add value

• The process of creating a test plan adds value, but the document itself is of limited value• The value of a test plan is the ability to

communicate the set of ideas that guides the test project to different stakeholders


Simplicity• Simplicity is the state or quality of being

simple [2]

• There is a value in keeping the test plan simple which relates to the burden which the test plan puts on someone trying to explain or understand it

• We also want to limit the amount of effort we spend on creating the test plan, and minimize the amount of unnecessary artifacts created

• On Google and Microsoft there is a concept of a 10 minute test plan, which should in accordance with the name only take a short period of time to create [3]


Simplicity

Complicated

Simple

Cost of Simplicity

Value of Simplicity

Valu

e Gai

n


Dynamic Test Plan• The key to reducing the size of the test plan is

to remove anything that cannot change over time

• Static information can be moved from a test plan to a Test Strategy or a Test Logistics document

• Anything that does not add value to any stakeholder should also be removed


Content of a Test Plan• Test Ownership

• System Configuration Under Test

• Definition of Done

• System Risk Matrix

• Test Activities

• Test Schedule

• Self-Learning Loop (SLL)


Overview

Identify Risks

Plan Test Activities

Self-Learning Loop

Ownership - Test Configuration - Definition of Done - Activity Description


Test Ownership• Which person is responsible for performing

what tests?

• This could be done on a overview scope level, for different test charters, or for specific test cases

• A test ownership for a specific person or group could be defined in the following ways:• 250 specific performance test cases

• All performance tests

• All or specific charters related to performance

• Performance

• Running performance tests during a specific time period or project phase


System Configuration Under Test• Most systems can be configured in many

different ways

• It is necessary to describe the configuration under test in order to• Reproduce tests

• Write bug reports

• Divide test ownership between different configurations

• Let all testers know what to test on

• This can change over time depending on the configurability of the system


Definition of Done• It is necessary to have a Definition of Done [4]

for different phases of the test project

• This is used to define when testing stops as implied by the name

• Definition of Done should include:• Expected System Quality

• Expected Test Coverage

• Expected Risk Level

• Formal Documentation Needed

• Definition of Done can change over time dependant on stakeholder needs


System Risk Matrix• What risks threaten the system from reaching the

definition of done?

• These risks change continuously over the life cycle of the project

• Risks can include (but are not limited to):• Software changes

• New hardware builds

• Software delta between configurations

• Hardware delta between configurations

• Unknown system dependencies

• Unknown environment dependencies

• Changed stakeholder focus


Risk Quantification• The Risk Matrix needs to be filled quantifiable

risks

• There are many different models available• Severity - Occurrence - Detection [5] is one

• Low risk, Medium risk, High risk, is another

• If a system has many configurations, each configuration needs a column in the risk matrix

• Risks most likely also need to be categorized in a good way to make the matrix usable


Test Activities• All different test activities need to be explained

in the test plan

• The scope of these test activities should not be static, since it is seldom efficient to run static activities

• Different activities can be added and removed during the life cycle of the project


Test Schedule• A schedule or plan of when each test activity

should be executed in time

• The schedule should be dynamic [6] and based on input from what is happening in the project, and what has happened during previous test activities


Self-Learning Loop (SLL)• It is critical to describe how the test plan will

evolve over the project life cycle, and then continuously document what we have learned from this feedback loop

• Continuous improvement of gaps in test scope

• Continuous improvement of understanding of system dependencies


Conclusion• Test plans should add value and not include

any waste

• Test plans should be dynamic – static parts should be included in other documents which can be reused between projects, or not documented at all

• These 7 bullets help include critical parts in the test plan, and not introduce unnecessary waste


References[1] A question about test strategyhttp://www.satisfice.com/blog/archives/63

[2] Simplicityhttp://en.wikipedia.org/wiki/Simplicity

[3] 10 Minute Test Planhttp://googletesting.blogspot.se/2011/09/10-minute-test-plan.html

[4] Definition of Donehttps://www.scrum.org/Resources/Scrum-Glossary/Definition-of-Done

[5] Severity, Occurrence, and Detection Criteria for Process FMEAhttp://www.fmeainfocentre.com/guides/ProcessPktNewRatings.pdf

[6] Dynamic Test Planshttp://www.slideshare.net/JohanHoberg/dynamic-test-plans

http://www.satisfice.com/blog/archives/63

http://en.wikipedia.org/wiki/Simplicity

http://googletesting.blogspot.se/2011/09/10-minute-test-plan.html

https://www.scrum.org/Resources/Scrum-Glossary/Definition-of-Done

http://www.fmeainfocentre.com/guides/ProcessPktNewRatings.pdf

http://www.slideshare.net/JohanHoberg/dynamic-test-plans

test plan simplicity

Technology