law of minor release - more bugs -3mu better software...

Law of Minor Release

More Bugs =⇒ Better Software Quality

Audris Mo kus

Avaya Labs Resear h

211 Mt Airy Rd

Basking Ridge, NJ 07920

audris�avaya. om

Aug 20, 2013

Motivation

Are there fundamental time-lag relationships among software produ tion

fa tors (Laws of Software Evolution)?

Can they be harnessed to improve software development?

A. Mo kus (Avaya Labs Resear h) Law of Minor Release Aug 2013 2 / 38

Studies of Software Evolution

Fo us on long-term trends in, e.g., software size

◮Su h trends are aused by non-software fa tors

◮World e onomy

◮Business pra ti es

◮Te hnology hange

◮Not lear how to use in pra ti e


Studies of Software Evolution

Fo us on long-term trends in, e.g., software size

◮Su h trends are aused by non-software fa tors

◮World e onomy

◮Business pra ti es

◮Te hnology hange

◮Not lear how to use in pra ti e

Investigating SW evolution by observing only software

≡divining reality from shadows on a ave wall


Proposed Solution

◮Investigate short-term and repeating relationships with a lear

me hanism originating from the way software is reated and used

◮Use information from outside software development ave

◮Answer pra ti al questions

◮Can we ompare quality among releases to evaluate the e�e tiveness of

QA pra ti es?

◮Can quality be approximated with easy-to-obtain measures, e,g., defe t

density?


Approa h

◮Start from lear assumptions

◮Observe fundamental relationships

◮Validate

◮Build more omplex propositions using validated relationships

De�ne: Bug

A user-observed (and reported) program behavior (e.g., failure) that results

in a ode hange.

De�ne: A tion Will Introdu e a Bug

A tion will in rease the han es of a Bug o urring in the future.


First Fundamental Law of Software Evolution

Formulation

Code hange will introdu e bugs

Me hanism

◮New ode has defe ts

◮New ode exer ises existing ode di�erently

◮Program behavior hanges

Note: platform hanges ause ode hanges

Eviden e

◮New releases bring new bugs

◮Model: a business-driven feature hange

=⇒ N ∼ Poisson(λ) �xes with delay T ∼ Exp(µ) [4℄


Model predi tion for one release

2001.0 2001.5 2002.0 2002.5

010

2030

4050

Calendar Weeks

Wee

kly M

Rs (P

erso

n W

eeks

)

New feature MRsActual Repair MRsPredicted Repair MRs


Model predi tion for 11 releases (using earlier release)

Calendar Weeks

Wee

kly M

Rs (P

erso

n Wee

ks)

010203040

1994 1996 1998 2000 2002

r1 r7

r2

010203040

r80

10203040

r3 r9

r4

010203040

r100

10203040

r5 r11

r6New feature MRsActual Repair MRsPredicted Repair MRs


Corollary 1: Need to Normalize by Change to Obtain Quality

How to normalize by hange?

◮divide by the number of pre-release MRs

◮divide by the LOC added or hanged

Hypothesis 1

In rease ↑ in the number of ustomer-found defe ts per

pre-release MR (a simple-to-obtain measure) affe ts users'

per eption of software quality negatively ↓


Qualitative eviden e: No

How to ompare software releases?

�we tried to improve quality: get most experien ed team members to

test, ode inspe tions, root ause analysis, ...�

�Did it work? I.e., is this release better than previous one?�

Everyone uses defe t density (e.g., ustomer reported defe ts per 1000

hanges or lines of ode), but �it does not re�e t the feedba k from

ustomers.�


Let's Peek Outside the Software Development Cave


Does the in rease in

the number of users and

the amount of usage

introdu e bugs?


Se ond Fundamental Law of Software Evolution

Formulation

Deploying to more users

will introdu e bugs

Me hanism

◮New use pro�les

◮Di�erent environments

Eviden e

MRs p

er We

ek (P

erson

Mon

ths)

Post Release

0

5

10

15

20

25

30 V 5.6 V 6.0

Release with no users

has no bugs


Third Fundamental Law of Software Evolution

Formulation

Longer (and heavier) use will introdu e bugs

Me hanism

◮New inputs and use ases are

en ountered over longer periods

◮More extreme environmental onditions

happen over longer periods

Eviden e

◮Bugs tend to be en ountered even after year(s) of usage

◮See Commandments below


Third Fundamental Law of Software Evolution

Formulation

Longer (and heavier) use will introdu e bugs

Me hanism

◮New inputs and use ases are

en ountered over longer periods

◮More extreme environmental onditions

happen over longer periods

Eviden e

◮Bugs tend to be en ountered even after year(s) of usage

◮See Commandments below

Does every user and every year of usage introdu e the same

number of bugs?


Commandment 1: Don't Install Right After the Release Date

Formulation

Users who install lose to the release date will

introdu e more bugs

Me hanism

◮Later users get builds with pat hes

◮Servi es team understands better how to install/ on�gure properly

◮Workarounds for many issues are dis overed

Eviden e

0.0 0.2 0.4 0.6 0.8 1.0

0.0

Fraction of customers observing SW issue

Time (years) between launch and deployment

Frac

tion

◮Quality ↑ with time after the laun h,

and is an order of magnitude better

one year later [5℄


Commandment 2: Don't Pani After Install/Upgrade

Formulation

A user will introdu e more bugs lose to their

install/upgrade date

Me hanism

◮Software is not hardware: parts do not wear o�

◮Mis on�guration or in ompatibility with the environment

Eviden e

1st month from install Three months from install

0.0

0.2

0.4

0.6

0.8

1.0

Frac

tion

of C

usto

mer

Issu

es E

ncou

nter

ed

◮Two thirds of ustomer issues (leading

to a software �x) are reported within

three months of install

◮Sample: 87 release/produ t

ombinations


Corollary 1: Customer Quality

Formulation

Software release quality from users perspe tive is the fra tion of:

◮The number of ustomers who report a bug shortly after the

installation over

◮The number of ustomers who install soon after the release date (e.g.,

within seven months)


Corollary 1: Customer Quality

Formulation

Software release quality from users perspe tive is the fra tion of:

◮The number of ustomers who report a bug shortly after the

installation over

◮The number of ustomers who install soon after the release date (e.g.,

within seven months)

�We live or die by this measure�

VP for quality


Testing Hypothesis 1: Defe t Density Re�e ts Customer

Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

M Customer Defects Per Pre−GA MR



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC

Customer Defects Per Pre−GA MR% of custmrs with defect within 3m. of install



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse

better

worse



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse

better

worse

worse

better



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse

better

worse

worse

better

better

worse



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse

better

worse

worse

better

better

worse

better

worse

Perfe t anti- orrelation?!


Trying Another Produ t

MM

M M

M

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Qua

ntity

C

C

C

CC

r5 r5.1 r6.0 r7.0 r7.1

MC

Customer Defects Per Pre−GA MRCustomer Defects/Installed System

Perfe t anti- orrelation again?!


Why ustomers like high defe t density?



Customers don't are about defe t density

◮Most ustomers try to avoid bugs

◮By not jumping to a major dot zero release

◮By not installing immediately when new release is available

Software salesmen don't are about defe t density

◮They want their ustomers to avoid bugs

◮By warning about produ ts that are likely to ause problems

Software support people don't are about defe t density

◮They want their ustomers to report as few problems as possible

◮By delaying wide installation of new releases


Lemma 1: Major Releases Have Few Customers

Minor releases have two to �ve times more ustomers

12

34

5X

tim

es m

ore

cu

sto

me

rs

Note: based on 38 major and 49 minor releases in 22 produ ts


Commandment 3

Thou Shell Have a Constant Rate of Customer Issues

Me hanism

◮The only thing ustomers like less than a Bug is


Commandment 3


Me hanism


◮The bug that does not get �xed for a long time


Commandment 3


Me hanism


◮The bug that does not get �xed for a long time

◮Team handling ustomer issues an not expand and ollapse

instantaneously and has limited throughput

Eviden e

2011−07 2011−12 2012−05 2012−10 2013−03

Numb

er of n

ew iss

ues for

one p

roduct

050

100150

Monthly number of

new ustomer issues is

relatively onstant


Law of Minor Release

Formulation

Minor releases have high defe t density but low han es

that any given ustomer will observe a defe t

De�nition

Major Releases Have More Code Change

Me hanism


Dis ussion

◮There exist Laws of Software Evolution, but

◮Fo us on short-term, repeating relationships with a lear me hanism

◮Look outside SW ave to observe them

◮Chose pra ti al questions

◮Pra ti e hints

◮Development pro ess view does not represent ustomer views

◮Maintenan e � the most important quality improvement a tivity


Referen es I

Audris Mo kus.

Empiri al estimates of software availability of deployed systems.

In 2006 International Symposium on Empiri al Software Engineering, pages 222�231, Rio de Janeiro, Brazil,

September 21-22 2006. ACM Press.

Audris Mo kus.

Organizational volatility and its e�e ts on software defe ts.

In ACM SIGSOFT / FSE, pages 117�126, Santa Fe, New Mexi o, November 7�11 2010.

Audris Mo kus and David Weiss.

Interval quality: Relating ustomer-per eived quality to pro ess quality.

In 2008 International Conferen e on Software Engineering, pages 733�740, Leipzig, Germany, May 10�18

2008. ACM Press.

Audris Mo kus, David M. Weiss, and Ping Zhang.

Understanding and predi ting e�ort in software proje ts.

In 2003 International Conferen e on Software Engineering, pages 274�284, Portland, Oregon, May 3-10 2003.

ACM Press.

Audris Mo kus, Ping Zhang, and Paul Li.

Drivers for ustomer per eived software quality.

In ICSE 2005, pages 225�233, St Louis, Missouri, May 2005. ACM Press.


Abstra t I

Traditionally software evolution models onsider long-term trends, but at su h s ales the primary driving fa tors are

extrinsi to software: the hanges in te hnology lands ape or business environment. However, shorter software

y les re ur in a predi table manner and an be best explained by a ustomer-provider equilibrium that leads to an

apparent paradox of software quality: the best quality releases have the most defe ts. In OSS and in ommer ial

produ ts ustomers/end users are riti al ontributors to software quality improvement: they dis over and report

defe ts that an not (or are too ostly to) be dis overed otherwise. As new fun tionality is delivered in major

releases, reliability ons ious ustomers typi ally stay on the sidelines until the se ond minor release with properly

working features, bug �xes, and stability improvements arrive. The major releases, thus, are used by fewer ustomers

and, onsequently, have fewer ustomer-reported issues. Understanding su h predi table software y les and the

me hanisms underlying them is essential for e�e tive software quality improvement and ustomer satisfa tion.


Audris Mo kus

Avaya Labs Resear h

233 Mt. Airy Road

Basking Ridge, NJ 07920

ph: +1 908 696 5608, fax:+1 908 696 5402

http://mo kus.org, mailto:audris�mo kus.org

Audris Mo kus is interested in quantifying, modeling, and improving software development. He

designs data mining methods to summarize and augment software hange data, intera tive

visualization te hniques to inspe t, present, and ontrol the development pro ess, and statisti al

models and optimization te hniques to understand the relationships among people,

organizations, and hara teristi s of a software produ t. Audris Mo kus re eived B.S. and M.S.

in Applied Mathemati s from Mos ow Institute of Physi s and Te hnology in 1988. In 1991 he

re eived M.S. and in 1994 he re eived Ph.D. in Statisti s from Carnegie Mellon University. He

works at Avaya Labs Resear h. Previously he worked in the Software Produ tion Resear h

Department of Bell Labs.


law of minor release - more bugs -3mu better software...

Documents