5. bug reporting - tunitie21201/s2011/luennot/ohj-3066_2011_209...5.bug reporting bug reporting is...

Department of Software Systems209

5. Bug reporting

Bug reporting is in practice the most important form of communication between testers and developers. Although the goals are usually clear, good reporting is not easy to accomplish. In many cases this is just an attitude problem.

Mika Katara: Software Testing, 2011


• Based on ”Bug Advocacy”, Cem Kaner, Quality Week 2002.• If the software is targeted to mass markets, most of the errors

found by the users have already been found in actual testing – why have they not been fixed?

• Bug report is a tool whose purpose is to convince the organization that money and time should be spend for fixing the bug



• If the purpose of testing is to find bugs, then bug reports are the primary results the tester provides– They are results that are displayed outside of the testing team

and for which the testers are remembered– The best tester is not the one who finds the most bugs, but the

one who gets most errors fixed• Because there is never enough time, a tester has to ”sell” the

bug to the developer, so that the developer would use his/her time to fix it

• The selling is based on two goals– The developer has to be motivated to fix the bug– The counter arguments and explanations of the developer have

to be countered by the tester


Department of Software Systems

What Makes the Developer Want to Fix the Bug?

• It looks really bad• It looks like an interesting problem• It affects many people• Its repetition is trivial• It makes us look bad, or the corresponding bug has made our

competitors look bad• The management wants the bug to be fixed• The programmer wants to do a favor to the tester and other

personal reasons• Appealing to professionalism, etc.

Mika Katara: Software Testing, 2011 212


What Makes the Developer Resist the Fixing of the Bug?

• The developer cannot repeat the bug• The repetition of the bug is not straightforward• The repetition is not possible with this information and getting

more information requires a lot of work• The developer does not understand the bug report• Unrealistic bug• Fixing requires a lot of work• Fixing is risky• There is no foreseeable effect for the user• Not important; nobody cares if this does not work



• It’s not a bug, it’s a feature• The management does not care about bugs like this• The developer does not like the tester or does not trust him,

or other personal reasons



How to Motivate to Fix a Bug?

• When tester notices a failure, it is just a symptom, the error itself has not been yet revealed

• Perhaps this failure is not the best possible example to show the effects this error causes

• Thus, more work needs to be done before writing a bug report to point out that the error is more severe and common than it looks at the first glimpse



Finding out the Severity

• When a tester has found a failure, the software has been driven to a vulnerable state

• When continuing testing from this state, the true severity of the error can been found out– By changing control (first A, then B or the other way around) – By changing the values of options and settings– By changing software and hardware configuration– Etc.



• Is it a new or an old bug?– If it is an old bug, the motivation for fixing is not great unless it

has caused complaints from the customers– People take new bugs more seriously– If it is an old bug, you might find new ways to repeat the bug in a

new version of the software• Then it becomes a new problem

– These points are emphasized during the maintenance of the program, when the purpose of a new version is to fix the most critical bugs

– By cleaning up the bug database, important information maybe lost!



• Finding out how common the bug is– Find dependencies from the configuration

• If the error cannot be repeated in the developer's environment, the motivation to fix the bug will be hard to achieve

• The bug report must describe the environments where bug appears

– Try to generalize the extreme cases• Boundary value analysis helps to search for the problem

– Negative testing, boundaries, errorneous inputs• When the bug is found you don’t have to settle just for the

boundary values anymore



• The tester has to report also the bugs that cannot be repeated– If the report is accurate enough, the programmer may be able to find out

the source of the error– When you notice that you are unable to repeat a bug, write down

everything that you remember from it immediately• If you are unsure what you did, tell it in the bug report

– It is also worthwhile to write down what you did before you noticed the anomaly

– Check from bug database, there might be something similar• This and other projects, customer complaints

– Try to change the timing of the program to slower and faster– Discuss with the developer and/or read the code– The bug that cannot be repeated is an error of a tester, you should learn

your lesson



• A recipe for a good bug report– Good, descriptive title helps to find the report from the database– Describe how you were able to repeat the error, include all the

steps– Describe what the steps were, and which steps are important

and which not for repetition of the bug and how the test results varied in different test runs

– Analyze the problem and tell how it can be repeated with a smaller numbers of steps

– The report has to be easy to understand– The tone has to be neutral– Only one bug per report– If a file is needed for the repetition, attach it to the report (or a

pointer) Mika Katara: Software Testing, 2011


The template for a bug report

• The description of the problem– One line description / title– The most important field in a bug report

• The management reads this when they want to know which bugs still remain in the program

• A number identifying the report• Who reports• Date of creation of the report• Name of the software or a component• Release and version number• Test configuration(s)



• Incident type– Programming error, design error, document inconsistency,

proposal, question• Is it repeatable

– Yes, no, sometimes, I don’t know: If a tester says that a bug can be repeated and the developer disagrees, the tester has to repeat the bug in the presence of the developer

• Severity• Priority

– Programmer / project manager / error manager fills this• The impact on the customers

– For example, technical support fills this



• Keywords• The description of the problem and how to repeat it

– Step by step• A recommended fix• Who is in charge

– Filled by the project manager / error manager• Status

– Tester fills this (at least initially): open, closed, dumped• Decision / Resolution

– Project manager owns this field– Waits, fixed, cannot be repeated, delayed fixing, works as

designed, need more information, duplicate, withdrawn, etc.Mika Katara: Software Testing, 2011


• The version number of a build that corresponds to the decision

• The maker of the decision– Programmer, project manager, error manager, tester

• Tester of the repair– If a tester finds a bug, he/she also tests the fix

• The version history of the bug report• Free form comments

– Remember neutral tone



6. Measuring software

Nothing is permanent in software development except change. With measurements, educated guesses can be made about the current state of the project and what should be changed and to which direction. On the other hand, product metrics can help in targeting testing to the most error-prone parts.



Metrics

• Based mainly on [Haikala&Märijärvi 06• What do you want to measure, what information is needed?• Do you need product metrics, project metrics or both? Or

would you like to measure something else?– The more the better?

• Some quantities cannot be measured accurately, such as the number of bugs left, but estimates can be given– If the number of the bugs left could be calculated, they would

probably be easy to remove too– Some estimates are better than others



• Attitude problems are often encountered in the measurement– In software projects there is always a danger that measurement

is directed at a person and his/her performance– Everybody should know how the results will be used

• Metrics have been used in testing quite widely– Test coverage– Targeting testing to the most error-prone parts of the software

• Code, which is complex, is often also error-prone



Start with the Simple Metrics

• With test metrics, it is usually worthwhile to start with some simple ones that seem trustworthy and collect them into a metrics suite– For instance, if you have means to measure if all requirements

have been covered at least on some level of detail, it should be a good starting point

– A mix of complementary requirements/specifications based and code-level testing metrics maybe useful useful

– Instead of small changes in the measured values, more interesting are big sudden changes and trends (are we getting better or worse)

– There is no silver bullet in metrics either – Change the metrics suite in small steps, trying to keep track of

the effectsMika Katara: Software Testing, 2011 228


Metrics vs. Risks

• Is there some relation with your metrics suite and the product or project risks?– Test your metrics suite by inventing scenarios where some

important risk realizes in the product or in the project to see if your metrics clearly reveal it

– If not, could you improve the suite to better detect such problems?

• Question the metrics that have been often wrong: what does the pass/fail rate really tell?

• Could you complement that information with some requirements/specifications/code-level coverage metrics, for instance?



Plan your Measurements Carefully

• A measurement plan is often needed in large projects/organizations: – why to measure– what to measure– who participates in the measurements– which parts of the system are measured– when to measure– how to collect and analyze the information?



Requirements for the Metrics

• Measurement result’s relation to business– The grounds for measurement– Shared understanding of why the metric is collected and how it

can and should be used• Ease of deployment

– Is a ramp up project needed?• Understandability

– What are the reasons for the changing values• Selectivity

– There must be a relation between the results and the target of the measurement

• If the value of the metric changes, it is known what has changed in the target



• Objectivity– The results do not depend on the person etc. doing the measurement

• Reliability– Accuracy, repeatability, a metric is not misused by accident

• Cost-effectiveness– The costs of measurement compared to the gain

• Reusability



S-Curve

• An example of the cumulative error curve of the system testing [Haikala&Märijärvi 06]:

• Move on to the acceptance testing when the error curve normalizes

• Unfortunately, the resources needed for this are very hard to estimate beforehand

found

Febr

uary

Num

ber

of e

rror

s

Janu

ary

Mar

ch

May

April

fixed

fixed



•

Are These Good Progress Metrics?

time

Test

ed re

quire

men

ts Tests executed, estimated

Passed tests, estimated

Passed, actual

Executed, actual



Non-Functional and Negative Testing

• How to measure non-functional testing?• How to measure negative testing?• As far as the concept of test case is used, there should be no

fundamental difference• However, it is extremely easy to ignore non-functional and

negative testing if the focus is on testing functionality• Some types of testing may not be easy to measure in terms of

test cases– Usability, performance, security, etc.



Direct and Derived Metrics

• Basic metrics: tests executed, bugs found, requirements/specification coverage, code level coverage, size, complexity, effort spent, etc.

• Derived metrics: defect density, relation of good/bad bug fixes, relation of found vs. potentially founds detects (what percentage of the bugs that could have been found were actually found), etc.

• Process-dependent metrics: test case execution speed, automation level, bug lifespan in days, test planning effort, quality debt in agile development, etc.

• Generally, the problem is not in inventing new metrics, since there are plenty available, but in selecting the right ones for your needs



The Goal-Question-Metric method (Basili)

1. Develop a set of corporate, division and project business goals and associated measurement goals for productivity and quality

2. Generate questions that define those goals as completely as possible in a quantifiable way

3. Specify the measures needed to be collected to answer those questions and track process and product conformance to the goals

4. Develop mechanisms for data collection5. Collect, validate and analyze the data in real time to provide

feedback to projects for corrective action6. Analyze the data in a postmortem fashion to assess conformance

to the goals and to make recommendations for future improvements


Source: http://goldpractice.thedacs.com/practices/gqm/index.phpBasili, Victor R., “Using Measurement to Build Core Competencies in Software”, Seminar sponsored by Data and Analysis Center for Software, 2005


You Get What you Measure!

• How to fool metrics?– Managers get what they measure

• You can achieve 100% code coverage easier by ignoring the test results

• You can increase the level of automation easily by removing the manual tests

• You can become a very efficient tester overnight by getting access to the source code and planting bugs you uncover the next day

– If some incentives are associated with metrics, the temptation to fool them becomes an issue



Example Metrics


Figure: Timo Malm, VTT. Data origin: Capers Jones. Software quality in 2008: A survey of the state of the art.


System Level Coverage Metrics

• Traditional metrics:– Requirements coverage – how well each requirement has been

tested • Coverage by importance of requirements

– Functional coverage • Map the tests to program functions, even user interface

functions– Coverage by different units

• Components, modules, etc.



• Other possibilities– Risk coverage

• How product risks have been covered in testing, especially the major ones

– Coverage by risk associated with requirements/functions– Coverage by what is essential to user segments

• 70 % of requirements important to business users have been tested, 83 % of what are important to the home user

– Usage scenario / use case / user story coverage• Coverage metrics used at this level depend on the

development process – V-model or an agile process?



6.1 Code Level Coverage Metrics

• With code level coverage metrics it can be determined that not enough testing has been done– To determine that there has been enough testing is something

else• It is good to remember that missing code cannot be tested• Even though system level testers do not need to worry so

much about the code level coverage, it is good to know the strengths and weaknesses of the different metrics since they are used by developers to measure the adequacy of low level testing



• Good unit testing must be required from programming teams and subcontractors– Quality of tests is a better strategy than just looking into the

metrics • And tests are just one quality assurance method: code

reviews, using static analysis tools, etc. are also important• Unit level metrics are produced automatically by tools

integrated into unit testing and/or low level integration testing– Very easy to use as requirements for moving to the next phase

(quality gates)



• We explain the different kinds of coverage metrics by presenting the program to be tested with the flowchart notation

• In testing, we are interested in the behaviors of the program that correspond to different execution paths

• An execution path describes the control flow of the program from its start to the end

Example from[Haikala&Märijärvi 06]

compute results

initialization

students left

get information for the next

student

has taken the exam

set grade

update student

information

compute distribution of grades

end

no

yes

no

yes



Path coverage

• If the purpose is to cover as much as possible, it would be natural to require that all possible execution paths are covered– Problem I: Loops make the number of paths too large

• Many programs are intended to be executed in infinite loops ready to receive events from outside => infinite number of different execution paths

• Can only be reached inside one function, for instance– Problem II: All the paths found by searching through the program

code are not necessarily possible in the execution• For example some path through a function presumes a

certain parameter value, and the function may never be called with this parameter value



• How many different paths of execution does the following program have?– In other words, how many different ways are

there to start from the start node and end up in the end node?

• If there are n students, and each of them has two possibilities according to if they have participated in the exam or not: 2n

• If there is a course with 100 students, there are 2100 execution paths

• If it would be possible to break from the loop in the middle of the execution, there would be a lot more paths: 2n+2n-1+2n-2+…+21+20

compute results

initialization

students left


student

has taken the exam

set grade

update student

information


end

no

yes

no

yes



• Path coverage testing is a goal that cannot be reached in practice

• Even if it was reached, it would not guarantee absence of errors:– Maybe something has been implemented outside of the

requirements– Maybe some of the required paths are not implemented– Even if all possible paths were tested, the parameter values

might have been selected poorly, so that failures would not occur• For example if(5/x + 5 < 10)

– For path coverage it is enough that tests with values 0<x 1 and x>1, but value x=0 is not necessarily tested and error is not detected



Statement coverage

• If for every statement in a program there is at least one path in the group of execution paths (i.e. test cases) that executes the statement it is said that statement coverage has been reached – In other words, a suite of test cases

executes each statement at least once

compute results

initialization

students left


student

has taken the exam

set grade

update student

information


end

no

yes

no

yes



• If a program has 100 statements, of which test cases execute 63, the statement coverage is 63%

• 100% path coverage would imply 100% statement coverage



Branch/Decision coverage

• As noticed on earlier, the branching points are essential when defining test coverage

• Decision coverage requires that each decision gets both of its possible values when tested

• Decision coverage means that each arc in the flowchart has been traversed

a==0 &&b>0 false

true



• Decision coverage is stronger requirement than statement coverage:if(a==0 && b>0) {

a++;}– For statement coverage it is enough

to have one test case for example (a=0, b=1), but for decision coverage two are needed, for example (a=0, b=1) ja (a=1, b=1)

initialization

a==0 &&b>0

a++

end

false

true

foo



• If for example out of 100 possible decisions 54 has been made, it is said that decision coverage has reached 54%

• 100 % decision coverage implies 100% statement coverage with the following assumptions:– In the code there is at least one decision– The execution starts always from the beginning of the code (no

goto commands, etc.) – In addition, triggering of exceptions might cause all decisions to

be covered but not all the statements



Condition coverage

• In the previous example, decision coverage was reached with two test cases, in which the decision got both of its possible values

• The decision consisted of two Boolean valued conditions (a==0) and (b>0), from which the latter got only one of its possible values

• Boolean valued conditions differ from decisions in that they do not contain logical operators such as AND or OR

• Condition coverage: all conditions evaluate to true and false• Note! Condition coverage does not necessarily imply decision

coverage: in the example test cases (a=0, b=0) and (a=1, b=1) satisfy condition coverage, but not decision coverage

• If both condition and decision coverage is required the metric is called decision/condition coverage



Multiple condition coverage

• Multiple condition coverage requires that all the possible value combinations for decisions are to be covered

• If a decision consists of n conditions, at most 2n test cases are needed to achieve multiple condition coverage for this decision

• In the previous example multiple condition coverage is satisfied for example with test cases (a=0, b=0), (a=1, b=0), (a=0, b=1) and (a=1, b=1)



• 100% multiple condition coverage implies– 100% decision coverage– 100% condition coverage– 100% decision/condition coverage

• If a decision consist of conditions that rule each others out, for example (a<0 || a>4), then a combination where both conditions would be true is obviously not possible– Therefore our definition sticks with ”possible” combinations– Note! In many programming languages short-circuit evaluation

reduces the number of possible combinations



• Multiple condition coverage can also be defined so that instead of all combinations of one decision, we have to cover all the combinations of all conditions of the program/function– More coverage is gained this way

• From the test coverage tools, for example CTC++ sticks to the former definition of multiple condition coverage



6.2 Complexity metrics

• Complexity measure• Idea: testing is targeted to the most complex parts of the code

– Beneficial mainly in unit testing• Complexity of the unit is measured and it is tested more or

less based on the need• Based on static analysis of the structure of the code



LOC

• Lines of Code• Is used to measure the length of the code and also for

example productivity of a programmer (LOC/day) • Could the number of lines be used as a metric for complexity?

– The lengthier the function, the more complex?• Let’s assume that LOC is a good metric for complexity• Let’s take a program P implemented with language X that has

a length of 1000 LOC • Port the program to the language Y



• The new implementation has a length of 500 LOC• Was the complexity of the program reduced in half or is

language Y just more powerful than X?• Conclusion I: LOC is surely a good metric for the length of the

code, but a bad complexity metric (at least by itself) • Conclusion II: A good complexity metric is not sensitive to the

programming language used– In the example, it would have been possible to calculate a

constant scaling factor when porting from one language to another



Halstead’ metrics

• In the middle of 1970’s, Maurice Halstead presented the ”Software Science” theory (Elements of Software Science, New York, Elsevier North-Holland, 1977) and a group of metrics derived from it

• A program is defined to be a collection of tokens that are divided into operators and operands– Operators are for example reserved words, mathematical

operators, etc.• Basic metrics are:

u1 = Number of unique operatorsu2 = Number of unique operandsN1 = The total number of operatorsN2 = The total number of operands



• For example, C++ language statement ”a == b” consists of one operator (==) and two operands (a and b)

• With the basic metrics, one can define more complex metrics • For program P

– Length is N = N1 + N2

– Vocabulary isu = u1 + u2

– Volume isV = N x log2u



– Program level isL = V* / V, where V* is the volume of the smallest possible

implementation– The inverse of level is difficulty

D = 1 / L– The estimate on the amount of work needed for implementation

(or for understanding) isE = (u1 x N2 x N x log2u) / (2 x u2)

– Time needed for implementationT = E / 18 seconds



• In some studies a connection has been found between the value of N and the number of errors that a module contains

• However ”Software Science” theory has been repeatedly questioned– In the definitions of the metrics there are measurement theory

related problems• On the other hand, for example SEI’s maintainability index

uses the volume metric by Halstead



The McCabe’s cyclomatic number

• Based on Thomas J. McCabe’s, A Complexity Measure, IEEE Transactions on Software Engineering, SE-2(4), Dec. 1976

• A graph theoretical base for estimating the complexity and the need for testing

• Cyclomatic number for a graph v(G) G that has n nodes, e arcs and p connected components isv(G) = e – n + p

• In a strongly connected graph G (from every node there is a path to every other node), the cyclomatic number corresponds the maximum number of the linearly independent circuits– A circuit is a path whose start node is same as the end node



• McCabe applies this in a following way– Let’s present the structure of a program as a directed control

graph that has a single start node and a single end node– The nodes correspond to sequential statements in a program

and the arcs the flow of control– Let’s assume that

• There is a path to all nodes from the start node• From all the nodes there is a path to the end node



– If an extra arc from the end node to start node is added, the control graph becomes strongly connected

a

b c

e

d

f

12 3

7

6

45

9

10

8



• Often one function at a time is under study, so p = 1• On the other had an extra arc from the end node can be

ignored, if it is represented as a constant in the calculations• Instead of circuits we can now talk about paths from the start

node to the end node, so the formula is minimized tov(G) = e - n + 2

• If all decisions are binary, meaning there are at most two arcs leaving from every node, the formula is simplified even more (proof is not presented here) to v(G) = h + 1, where h is the number of binary decisions



• Here we assume that the conditions connected with AND are separated to be their own decision, for example IF c1 AND c2 THEN IF c1 THEN IF c2 THEN

• If the decisions are not binary, like the CASE statement, they can be simulated with multiple IF statements



• Properties of v(G) – v(G) 1– G: has only one path if and only if v(G) = 1– v(G) is the maximum number of linearly independent paths,

meaning the number of basic paths– Adding sequential statements has no effect on v(G) – Adding a new branch increases the value of v(G) by one– v(G) depends only on the number and structure of the decisions

in G• In some studies a correlation between the number of errors

and the value of v(G) has been found



• The recommendation is that every function has v(G) 10 – The functions that are more complex than this are probably hard

to test and to maintain• If necessary, the function is to be divided into two (or more)

simpler functions• According to McCabe, a function should be tested at least

with such test cases that cover the basic paths– There are many sets that cover basic paths

• The number of these basic paths is easy to calculate (v(G)), but coming up with the actual paths is not so simple without proper tools


5. bug reporting - tunitie21201/s2011/luennot/ohj-3066_2011_209...5.bug reporting bug reporting is...

Documents