Stephen Blower

@badbud65

QA is Broken.Fix It!

"In every chain of reasoning, the evidence of the last conclusion can be no greater than that of the weakest link of the chain, whatever may be the strength of the rest.“

Thomas Reid's ”Essays on the Intellectual Powers of Man” 1786

How many different development/test

methodologies are in existence?

Question:

Spiral Model of Software Development (1986) 

Rational Unified Process (1994)V-Model (1982)

Waterfall (1984)

Scrum (1995)

Crystal Clear (1996)

Extreme Programming (1996)

Adaptive Software Development (1995)

Feature Driven Development (1995)

Dynamic Systems Development Method (1995)

Agile Manifesto Published (2001)

Test Driven Development (2003)Rapid Application Development (2004)

Goal-Driven Software Development (2006)Behaviour Driven Development (2009)

IntroductionNo one right way

We all desire perfection but, the road to success is often misleading with self confessed Oracles.

Fantasy Reality

No one Development Team is the SameNumerous Tools

Numerous Methodologies

Diverse Levels of Experience and Skill Sets

Wide ranging levels of Enthusiasm and Energy

Optimistic and Pessimistic Attitudes Towards Change

Strict Process Driven or Flexible and Informal Practices

Personal Ownership or Strict Hierarchical Structures

Development Processes Should be about the People

Not How to Control the People

The ideas I would like you to take away from this talk

• Software development hasn’t advanced as far as people like to believe

• Too many ideas, little consensus and no proven methodologies

• No one software development methodology is the holy grail

• Change is difficult, move forward slowly and throw away broken processes

What is QA and what do I mean by Broken?Quality Assurance?

Dictionary Definition:

Quality, defines a “Certain Level of Excellence”.

“Excellence” meaning “Extremely Good”

“Assurance” means “Keeping Promises”

So Quality Assurance means:

Keeping Promises to Supply Products or Services to a Certain Level of Excellence.

~Johanna Rothman put this really well in a talk, she said that when testers tell her that they do “Quality Assurance", she asks questions like these:

1. Do testers have the authority and cash to provide training for programmers who need it?

2. Do testers have the authority to settle customer complaints? Or to drive the handling of customer complaints?

3. Do testers have the ability and authority to fix bugs?

4. Do testers have the ability and authority to either write or rewrite the user manuals?

5. Do testers have the ability to study customer needs and design the product accordingly?

If not, the quality of the product and of the complaining customer's experience in dealing with it, are not under the testers' control.

This exert is from a Paper Called: The On-going Revolution in Software Testing: Cem Kaner

We are all doing it wrong, because no one knows how to do it right, including me.

A Brief History of Software

DevelopmentHow did we get into this mess?

A Brief History of Software Development Practices

The Agile™ Manifesto states:

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value: Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.

When was agile first used?

Quote: From: Iterative and Incremental Development: A Brief History

Craig Larman of Valtech & Victor R. Basili of University of Maryland

Gerald M. Weinberg:

“We were doing incremental development as early as 1957, in Los Angeles, under the direction of Bernie Dimsdale [at IBM’s Service Bureau Corporation]. He was a colleague of John von Neumann, so perhaps he learned it there, or assumed it as totally natural. I do remember Herb Jacobs (primarily, though we all participated) developing a large simulation for Motorola, where the technique used was, as far as I can tell, indistinguishable from XP.”

XP wasn’t published until 1996, nearly a full 40 years later.

1957! Yes 1957!

Don’t Blame Royce, Blame the US Military.

Winston W Royce: “Risky and Invites Failure

1984 : American Department of Defense for System Software Development created DoD STD 2167

It promoted a one-pass document-driven waterfall process.

Anecdotally: US Military just took this picture to develop the Waterfall model

Why did Waterfall gain popularity?

HierarchicalTop down structured approach

Strict controlsCreates illusions of control

Now we have 101 ways of doing the same thing:

Developing Software

Software development is still in its teens and as such there is not any empirical evidenced based approach to software testing.

So which methodology is the right one?

SDLC 100 Success Secrets - Jeremy Lewis Typical Snake Oil - Not one of the 100 (Supposedly) Asked Questions are anything you couldn’t answer yourself if you have a modicum of experience within software development.

Common Failings•Not knowing what a PROBLEM is

•Not knowing how to change

•Not knowing what change looks like

•Not understanding the model/s chosen fully

•Not spending the time to implement the change

•No real desire for the change, just a general feeling

•Choosing the wrong model or choosing one model

•Not having a detailed delivery plan

•Not having clearly defined goals

“A goal without a plan is just a wish.”

Antoine de Saint Exupéry

Assessing the Current

SituationAssess for Success

What is a Problem?Something is not working? Yes?

Not strictly true

Who says it’s not working?

In what way is it not working?

How do you make a problem go away?

a) Change the differenceb) Change the perception

c) Change the desire

Jerry Weinberg calls people who rush to solve problems as a "solution problemmer". Not a problem solver, but a solution problemmer.

Basically: "If our solution doesn't work, *you should change your problem*.“

“A difference between what is perceived and what is desired.”

More specifically:

A difference /according to some person/ between what is perceived /by some person/ and what is desired /by some person/.

“An undesirable situation that is significant to and maybe solvable by some agent, though probably with some difficulty.”

Definitions and Stories, from and inspired on Skype conversation with Michael Bolton

Test - Serve the Business

Testing is a service role. Feel good about that. The service you provide is vital. Service implies clients—the people you serve. Your success is measured primarily by how well you serve your clients'

desires and best interests.

Project Managers

PMs are entitled to know your process and influence it. You serve the project manager by reporting your status on demand, reporting important problems fast, and not being a bottleneck to the project.

Document Writers/BAs

Like you, the people who write the documentation get incomplete information. You can help them understand how the product really works. Writers can help you, too. As they do their research on the product, they will learn things that you don't know.

Technical Support

Whatever problems are left in the product they become a burden for the people who provide technical support. You serve support by alerting them to aspects of the product that may trouble the user. If you work with them, they can help make the case that a bug should be fixed.

Senior Management and Share Holders

You serve the business. That's why you must be careful not to sound or act like a quality fanatic instead of a reasonable person. Express test status reports in crisp, operational terms, so that executives feel they have a basis on which to make decisions.

Marketing/Sales

Marketing & Sales need to know whether anything in the product is inconsistent with the key benefits the product is supposed to provide to customers. A bug that seems minor to others might be critical to marketing & sales.

End User

In your heart, you serve the people who will make use of the product. Their satisfaction is in the best interests of your project, of course. But there is also a special satisfaction that goes with being the primary user advocate on the project team.

Developers

You make the developer's job easier by providing good bug reports, as soon as possible. Strive to know your craft and know the product so you don't waste the developer’s time with mistaken or frivolous reports.

Lessons Learned in Software Testing: Section 1.3. Lesson 3

Talk to Everyone

“Talk to someone about themselves and they'll listen for hours.” 

Brush up your social skills

Understand what is wrong

Access morale

Gather information uncritically

Understand desire to change

Gain constructive feedback

“Any fool can criticize, condemn and complain--and most do.”

Management opinions

Stay Neutral

Don’t take sides

Be open to discussion

Constantly invite feedback

Discuss opinions rigorously

All opinions are weighed equally

At this stage, they are

Everyone has an opinion

Be able to defend your arguments in a rational way. Otherwise, all you have is an opinion. - Marilyn vos Savant

To gain a fuller understanding

Get you hands dirty and test

Get familiar with the tools

Get familiar with the processes

This is essential

To gain respect

To gain direct experiences

To understand the products

Also get to understand your competitors

33% No – 30% Probably and 37% Definitely – That’s Scary

Delving Deeper• How do teams work together? Are they working closely or are they

siloed? How does information flow between teams? Is it managed in any way? Are people expected to source the information themselves or does the project manager ensure the right people are kept in the loop at the right time?

• Are people in teams listened to? Does one person in a team feel they are not listened to yet another has a totally different opinion? Why is that?

• How have changes been implemented previously? Did they work, if so how did they work? If not, why not?

Delving Even Deeper• What is the current process? Is it documented? Does everyone know what the

process is?

• Is there a roadmap? Do people know what is happening in two weeks’ time?

• Does test know when the next product is coming in to test? Are test consulted about the test schedule? Is there a test schedule?

• Are test involved at the design stage? Are test involved in any stage of the development process? Is test only involved when the product is code complete?

Creating you plan for change

Ideas are now growing roots

Ideas for change based on:

Experience with previous change

Practical hands on experience

Everyone's opinions

Personal experience

Change is now inevitable

Not just an idea

There’s a time line

This shits now getting real, baby.

Argue and discuss changes:

Convincingly

Collaboratively

Respectfully

Understand objections:

Is it practical

Is it appropriate

Be prepared to:

Throw away plans

Change plans drastically

“No man is an Island” - John Donne 

James Bach

Summary: Assessing the Current Situation

• Talk to everyone and Listen to everyone• Do not weigh any option above any other• Stay neutral, be uncritical and stay positive• Continuously gain feedback• Gauge desire for change• Gain support and buy-in• Keep your enthusiasm levels high• Understand current processes, what works and what doesn’t• Understand how to communicate effectively• Learn to argue in a Socratic style• Interact directly, do the job of the testers• Start planning early• Create, modify, scrap plans and constantly evaluate plans

NOTE: I talk about measuring the success of new changes in the last section

Planning how to Deliver

ImprovementsPlan for Success

Think outside the box, while retaining what’s useful in the box

ALL! Changes for improvement identified

Don’t be fooled

Some ideas will be scraped

Some ideas won’t get buy-in

Continue to talk, discuss & collaborate

It’s not about you

It’s about improvement

Keep management in the loop

Create a realistic time line

They don’t like surprises

Don’t: “Over Promise and Under Deliver” or “Under Promise and Over Deliver”

Get buy-in

To agree with; to accept an idea as worthwhile.

To communicate a vision and create buy-in.

From the supporters

From opposes, this is essential

From management

From the ideas people

From the people who desire change

From the people who hate change

“Deep and sustainable change... requires changes in behavior among those who do not welcome the change.” ― Douglas B. Reeves

Process, Process, Process!

Avoid like the plague

Process changes are needed : Yes

Process for sake of process : No

Processes needed : Yes

But streamlined, not reams of docs

The won’t be read anyway

No Best Practices PLEASE!

“A “best practice” is an idea that a consultant thinks he can sell to a lot of people. There is no assurance that this idea has ever succeeded in practice, and certainly no implication that it has been empirically tested and found superior (best) to competing ideas under general conditions.”

Cem Kaner

Lets not call them Best Practices, lets call them Good Practices

Best: The highest quality, excellence, absolute qualifier, context independent

Practice: Habitual or customary performance

Therefore: Best Practice is: The highest quality of habitual performance with no context

Lloyd Roden Definition

Summary: Planning how to Deliver Improvements

• Keep talking• Get Buy-in from everyone and anyone you can• Be prepared to constantly evaluate plans for change• Collaborate• Keep your mind on the end game, Improving Test• Introduce small changes early• Always have you eye on the time line• Do not “Over Promise and Under Deliver” • Do not “Under Promise and Over Deliver” • Be realistic with your goals and time line• Don’t write numerous reams of documentation, they won’t be read• Don’t create numerous new processes, they will hate you for it• Don’t create ANY Best Practices• Get agreement

Inspire and Measure

Deliver for Success

Leadership comes in small acts as well as bold strokes - Carly Fiorina

Enable, Empower and Enthuse

Lessons Learned in Software Testing:Read it, don’t expect others to though

Lean Coffee sessions:Explore testing ideas

Pair Testing:Learn together, challenge

Challenge testing ideas:Challenge the status quo

Challenge, push and drive:Question, question, question

Encourage, nay demand feedback:Focus on those who don’t feedback

Praise with sincerity:Don’t praise for sake of praise

“Some men (sic) have thousands of reasons why they cannot do what they want to, when all they need is one reason why they can” ― Martha Graham

Measurement, ARGH!

Measure for improvement

Measure sparingly

Be aware of error margins

Don’t set targets to be achieved

Metrics give you information

It’s primarily about PEOPLE not metrics

Gain feedback on metrics

Ensure test are involved with metrics

Recommended Reading: Measuring and Managing Performance Organisations : Robert D Austin.

"When a measure becomes a target, it ceases to be a good measure“ - Dame Ann Marilyn Strathern

Did you take away these ideas from this talk?

• Software development hasn’t advanced as far as people like to believe

• Too many ideas, little consensus and no proven methodologies

• No one software development methodology is the holy grail

• Change is difficult, move forward slowly and throw away broken processes

Conclusion

• There’s no such thing as best

• There’s no such thing as perfection

• Don’t create process after process

• Don’t create any Best Practices

• Don’t create reams of documentation

• Change, challenge, and learn, together

• It’s about people, not how to control people

“If you meet the Buddha, kill him.” - Zen Master Linji – Blog: Adam Goucher on Heuristics

Questions?

He must be very ignorant for he answers every question he is asked - Voltaire