Supercharging Your

Bug Reports

Neil Studd (@neilstudd)

TestBash Workshop Day

Thursday 26th March 2015

About me

• Ten years of testing

• Working for acquired startups

• Senior/lead/mentorship positions

• Currently consulting in London

• Beta/freelance testing in spare time

• Facilitator, Weekend Testing Europe

This will be a bit like Weekend Testing…

• Two hours

• Too much content!

• Driven by you – we’ll “follow the energy”

• Link to resources at end of session

• It’s not a lecture – pitch in whenever you want!

• Share and compare experiences

About You!

Talking about… bugs (or problems, or defects)

• Bug: “Any problem with the product that might threaten its value” (Rapid Software Testing)

• …or “something that ‘bugs’ somebody who matters” (James Bach)

• Problem: “A difference between someone’s desires and the way things seem to them” (Jerry Weinberg)

• Or defect, or issue, or fault, or backlog item…

• Whatever you choose to call them, we’re going to focus on what happens between discovering one of them and it being fixed.

So… what’s a bug? (BBST Bug Advocacy)

Doesn’t match specs or requirements

Could embarrass the company Generates incorrect or confusing results

Doesn’t match documentation Reduces the chance of a sale Wastes the time of a user

Doesn’t do what the programmer intended (coding error)

Interferes with development of the product

Anything that would lead to a code change if reported

Doesn’t meet company standards Interferes with the testability of the product

Failure to meet reasonable expectations of a user

Doesn’t meet industry standards Interacts badly with other programs or components

Failure to meet reasonable expectations of a stakeholder

Does this stuff really matter?

• Most testers discover/report about bugs

• Very few testers have any formal training on bug reporting

• Testers have a wealth of shared experiences in this field, but we each log bugs in isolation

• Bug reports are like a sales tool for bugs. Need persuasive powers!

• Bug reports are often a tester’s primary delivered work product

• Reporting skills are vital for credibility

Credibility

We GAIN credibility by:

• Using clarity and accuracy in writing

• Pinpointing the exact steps needed for a subtle bug to occur

• Giving decision-makers enough information to cast a judgement

• Displaying professionalism - not creating a blame culture

• Owning up to our mistakes

We LOSE credibility by:

• Wasting a developer’s time

• Making accusatory statements

• Making false assumptions or inflating the status of a bug without reason

• Being the boy who cried “bug”

When a contentious bug arises – your credibility may directly affect the outcome!

Bug Advocacy = ‘Selling’ a problem

• Advocacy: “Public support for a particular cause or policy”

• Make a case for the fix; pick your battles

• Take ownership of the problem as a champion for quality

• Testers don’t “break the product” but are well-placed to push for fix

• Effective communication is key to good decision-making

• Motivate the buyer: Make them WANT to fix the bug

• Overcome/pre-empt objections, removing reasons for NOT fixing the bugs

What makes a good bug report?

• What sort of information do you include?

• What information do you omit?

• Think of adjectives which describe a good bug report.

• If it helps, think about your company’s bug-tracker:• What fields are mandatory?

• What fields are mandatory, but you wish they weren’t?

• What fields are missing that you’d like to see?

PEOPLE WORKing: Heuristic for reports

• P

• E

• O

• P

• L

• E

• Work

Presented by Michael Bolton at EuroSTAR 2014

Co-authors include James Bach & Alexei Barantsev

Problem

Example

Oracle

Polite

Literate

Extrapolation

Workaround

PEOPLE WORKing: Problem (1/2)

• Title/Summary

• A summary of what you’ve observed

• What happens, under what conditions

• Describe the relationship between perceived and desired

• Balancing length restrictions versus ease of comprehension

• Grabs attention, starts a conversation, serves as a call-to-action

• Searchability: “Where’s that bug for the homepage dropdown…?”:

“Expected” v “Desired”

PEOPLE WORKing: Problem (2/2)

Alan Page, “How We Test Software At Microsoft”:

• Title is used for review by feature/product triage team

• In the space of ~80chars, needs to accurately summarise the report

• “At Microsoft, many testers quickly fill out all fields of the bug report but take extra time to carefully word a title for their bug.”

• Program crash – too short

• When running many instances of the program at the same time, a crash occurs in one of the dialogs – both wordy and vague

• Program crash in Settings dialog box under low-memory conditions –specific, accurate, and tells enough of the story to understand the bug

PEOPLE WORKing: Example

• An illustration of the problem, as clearly and briefly as possible

• Narrow scope: Find critical variables, drop steps, subtle variations

• “What is the fastest, least effortful way to report this problem that still completely addresses what people need to know in order to get to it” – Michael Bolton

• If looking for the “fastest” report, consider how your reporting tool might be working against you (mandatory fields; fields where you add the same value every time)

• Commonly “Steps to reproduce” and/or screenshots

• Record information (steps, logs, screenshots) as soon as you see the issue. It might not be as reproducible as you think!

Mongoose or antelope? (Simon Tatham)

• Mongoose: Attacks frantically, because it’s better than nothing

• Antelope: Freezes, stops, thinks about the best course of action

• When a bug occurs… be the antelope!

PEOPLE WORKing: Oracle

• What can you apply to the situation to tell whether it’s a problem?

• Oracle: a fallible way to recognise a problem when it occurs in testing

• Consistency heuristic: FEW HICCUPPS (e.g. Comparable products)

• Our feelings can be heuristic triggers; emotions help us to reason

• Cite good oracles, which indicate a potential threat to product value

PEOPLE WORKing: Polite

• A bug report is usually bad news. Don’t kick them while they’re down.

• Avoid antagonising or apportioning blame

• Own the problem for the team (“we have a problem”)

• Polite also means not wasting time:• Don’t write in a way that’s hard to understand

• Don’t omit information which was obviously necessary

• Don’t waffle, get to the point, make the key facts easy to find

PEOPLE WORKing: Literate

• If you just have a title and an example, the natural reaction in many cases is “So what?” – why should anyone care about this bug?

• Tell the “story” of the problem:• Characters: The product, at least one person

• Build empathy for the person (they must matter, we must care about them)

• Reveal the problem (the problem must matter too)

• Develop a plot (the threat to value; use extrapolation)

PEOPLE WORKing: Extrapolation

• Is bug more serious or more general than first seems? (Fix chance ++)

• Once the bug has occurred, is system vulnerable to other problems?

• Cem Kaner – RIMGEA • Replicate• Isolate• Maximize (how could this be the biggest, baddest problem possible?)• Generalize (where else might this bug manifest in the code?)• Externalize (where else might this bug manifest in the wild?)• And say it clearly/dispassionately

• BUT be realistic – don’t exaggerate, provide salient facts

• Use safety language (e.g. can/could) but do it sparingly

PEOPLE WORKing: Workaround

• Is there an alternative solution which could lessen severity?

• Just provide information about it, it’s management’s decision as to whether it’s good enough

• Without a workaround, it could be a showstopper

PEOPLE WORKing: Heuristic for reports

• Problem

• Example

• Oracle

• Polite

• Literate

• Extrapolation

• Workaround

• Is this a good list?

• What might you add?

• Review some of your own bugs against these criteria

It’s logged – now how to get it fixed?

• Congratulations on logging a brilliant, eloquent, convincing report.

• Bug reporting is not just about bug reports!

• Triage stage: formal/informal, intra-team or “Bug Review Board”

• Brainstorm: What are the hurdles to actually getting a bug fixed?

OBSTACLE POSSIBLE SOLUTION

Can’t reproduce Better/clearer example, walkthrough, …

Don’t understand problem …?

Not enough time …?

…come up with more as a group

Invalid / “Won’t Fix” (1/2)

• “It’s not a bug, it’s a feature!” – use oracles

• “It’s unrealistic / nobody uses it in that way” – get data

• “Nobody’s reported that” – are you sure? Check with Support

• “It’s too hard / too risky to fix that” – goalposts shifts as release nears

• Have they understood the report? Were you clear/brief enough?

Invalid / “Won’t Fix” (2/2)

• Learn when to let go.

• If the people in charge are saying it’s not important, and your best information can’t convince them, it’s not your problem.

Precision

• Say exactly what you mean; clarify what you mean when you say it!• Particularly for product-specific / domain language. Get that wrong and

people might lose trust in whether you know what you’re talking about.

• Be aware of what’s being left open to interpretation.• “Mary HAD a little lamb”

• James Bach – “Two scoops of raisins”

• Write in a way that the problem is fully understood, so that the fix doesn’t just fix one small variation.

• If you’re speculating – say so, separate this from facts.

Precision, but also be brief…

The eventual fix was here all

along… everyone tuned out!

Cannot Reproduce (1/3)

• Bugs are rarely truly “random”, even something like this…

• Not random, but the circumstances under which it occurs is obtuse.

• (Debug, monitor value of x, you’d notice the pattern quickly.)

• The trick is to find ways to break down the pattern…

Cannot Reproduce: Finding the patterns… (2/3)

• Some emails fail to send

• Printer sometimes jams when printing

• Xerox machine doesn’t scan some documents correctly

• OpenOffice keeps refusing to print files

• My savegame file sometimes gets deleted

Cannot send email over 500 miles

Print this file; your printer will jam!

Xerox machine changes numbers on scanned documents

OpenOffice can’t print on Tuesdays

Savegame file deleted when PlayStation controller battery dies

Cannot Reproduce: Tips for Progressing (3/3)

• “Works on my machine!”

• Can YOU still reproduce it? (Did you give yourself enough info?)

• What are you doing which isn’t in the bug report? (Be obtuse!)

• What have THEY tried to reproduce it? (Pairing)

• Desktop: How do your environments differ? (Too clean? Too dirty?)

• Web: Browser? Version? Extensions? Dev Tools?

Duplicate (1/2)

• Why do duplicates happen?• Didn’t search

• No time to search

• Can’t search (repo doesn’t support it)

• Searched but didn’t find anything – know your system’s search grammar

• Search is key… so apply SEO techniques when creating content!• Keywords – consistent terminology

• Cross-referencing other issues (in text, or by linking bugs if system allows it)

• Errors in screenshots aren’t searchable; replicate it in text

Duplicate: Get error messages in text! (2/2)

…or type it! (If you have to. First few lines of stack trace + screenshot)

Formality

• Vary as appropriate

• “You are not a bureaucrat” (Michael Bolton)• Expected: Total field should update and display correct result

• Actual: Total field updates and displays incorrect result

• If you’re forced to complete meaningless mandatory fields, ask why

• Ideally as lightweight as possible

• Different team dynamics will operate at different levels

Formality – Ian MacKinnon“I think the best barometer for the culture of a company is in the formality of the bug reports.”

Formality - “View Source”

Going round the fixed / not-fixed loop

• Fixed > Rejected > Fixed > Rejected > Fixed > Done

• Michael Bolton, Christian Legget: “Boomerang” bugs

• My experience: “Bouncing” bugs (like a ball, the bounce gets smaller)

• How to avoid?• Dev-box testing – get at least a cursory hands-on before commit

• Understand why the developer believed it was fixed (environmental?)

• If it’s a regression then an automated check could help

• If it’s too hard to fix as desired, is there an alternative solution which could at least lessen the severity?

“We never get rid of problems… The best we can hope for is that the problems we substitute are less troublesome than the ones we ‘solve’.”- Jerry Weinberg, Are Your Lights On?

“Trivial” bugs

• Should you log them?

• Open discussion

Do you even need a report?

• Overhead of creating understandable report, waiting for time to pass, reviewing it, getting back into headspace…

• Consider: Post-its, mention-in-passing, tea!

• Investigate whether it really is small, or whether you think it is

• Take responsibility for the full stack of related activities; if you tell a dev it’s going to be “quick”, make sure it is (for them).

• A quick fix does not necessarily mean a quick test

• Respect your colleagues! “Quick fix” still has context-switch cost

Honing your bug-reporting skills

• Paid projects: uTest, TryMyUI

• Live bug bashes: The Defectives, hack days (be a “tester-for-hire”)

• Beta testing: Public betas for applications, websites, games

• Peer review: Within your team (gamify it?), Weekend Testing

Bug counting and metrics (1/2)

• The Daily WTF: “The Defect Black Market”

• “Don’t log that, we can’t fix it and it will make our stats look bad”

• If you didn’t log any bugs… what does that mean?

Bug counting and metrics (2/2)

• If you’re close to the team, you have better metrics available to you

• If you’re not close to the team, bug counts alone aren’t enough

Ask:

• “What are you actually trying to measure?”

• “How do you plan to add value with that information? To whom?”

• “What information might serve you better?”

• …What else?

“The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the right bugs fixed”- Cem Kaner

Bug counting and metrics

If all else fails…

• If there’s a culture of bugs building up, how can you fix it?

• “Cheese Day” (Blizzard)

• Beer/bacon bounties!

• To conclude: Your own experiences?

Whatever Works

• A means to an end – focus on whatever eases your path to the end

• How do you describe your techniques to newcomers?

• How do you know it’s working?

• How do you recognise what isn’t?

• Don’t change for the sake of it

Thanks for taking part!

Resources from today’s session:

http://blog.neiltest.com/testbash2015/

Twitter: @neilstudd

Email: neil@neilstudd.com