on the perception of software quality requirements during the project lifecycle

On the perception of software qualityrequirements during the project lifecycle

Neil A. Ernst, John Mylopoulos{nernst,jm}@cs.toronto.edu

University of Toronto

July 1, 2010

Ernst, Mylopoulos (University of Toronto) Perception of quality req. July 1, 2010 1 / 16

Motivation

On the perception of software quality requirements during the projectlifecycle

Throughout the lifecycle

Understand software quality requirements using an empirical,data mining approach

Quality requirements as universal requirements

Dataset: 8 Gnome OSS projects

Data / code available athttp://github.com/neilernst/gnome-quality-mining


http://github.com/neilernst/gnome-quality-mining

Specific research questions:

1 Test Lehman’s 7th Law1: system quality will appear to bedeclining unless rigorously maintained and adapted toenvironmental changes

2 Are high-level quality requirements (e.g., efficiency) of equalconcern in different projects?

3 Is this a viable mechanism for understanding the treatment ofquality requirements?

1Lehman, M.M., et al.: Metrics and laws of software evolution – the nineties view. In: International Software Metrics

Symposium pp. 20–32. Albuquerque, NM (1997)


Methodology

Generate corpora

start

Create signifiers(WN,ext)

Query corpora

Error analysis

Plot outcomes

and calculate stats

end

ISO9126, Wordnet, Boehm, McCall, KDE

Gnome projects: commits, bugs, mail

Precision/recall on random events


Datasets

Product Language Size (ksloc) Age (years) Type

Evolution C 313 10.75 MailNautilus C 108 10.75 File mgrMetacity C 66 7.5 Window mgr

Ekiga C++ 54 7 VOIPTotem C 49 6.33 Media

Deskbar Python 21 3.2 Widgets/UIEvince C 66 9.75 Doc viewer

Empathy C 55 1.5 IM


Signifiers and quality requirementsCross-project comparisons using a standardized taxonomy – ISO9126.Find quality concepts in project artifacts as signifier occurrences.Corpora: emails, bug reports, commit messages.Database of 900k ‘messages’ (total, all projects).

This menu is barely usable!

email


SignifiersDetect the appearance of quality concepts in the messages.Created signifier lists, a representative signifier thesaurus.Two lists: one based on Wordnet and ISO9126, the otherextended with domain knowledge (ext). E.g.,

Quality Signifiers

Maintainability testability changeability analyzability stability maintainabilitymaintain maintainable modularity modifiability understandabil-ity

Functionality security compliance accuracy interoperability suitability func-tional practicality functionality

Portability conformance adaptability replaceability installability portablemovableness movability portability

Efficiency “resource behaviour” “time behaviour” efficient efficiencyUsability operability understandability learnability useable usable ser-

viceable usefulness utility useableness usableness serviceable-ness serviceability usability

Reliability “fault tolerance” recoverability maturity reliable dependableresponsibleness responsibility reliableness reliability dependable-ness dependability


Observations1 Search for events2 Normalize for message volume3 Group by week

Event Quality

...By upgrading to a newer version of GNOME youcould receive bug fixes and new functionality.

None

There should be a feature added that allows you tokeep the current functionality for those on work-stations (automatic hot-sync) and then anotheroption that allows you to manually initiate .

Functionality

Steps to reproduce the crash: 1. Can’t reproducewith accuracy. Seemingly random. ....

Reliability,Functionality

These calls are just a waste of time for clientand server, and the Nautilus online storage viewis slowed down by this wastefulness.

Efficiency


Error analysis

1 Pick random messages; manually classify

2 Compare manual classification to machine classification

3 Precision: 80%, Recall: 51% (extended), 6% Wordnet

Signified quality Precision ext Precision WN

Usability 0.47 0.22Portability 0.11 0.20

Maintainability 0.22 0.31Reliability 0.15 0.19

Functionality 0.14 0.18Efficiency 0.16 0.07

Mean 0.21 0.20


Observations

RQ1: Lehman’s law

Is there a relation between events and project lifespan?

Looked at three possible explanations:I Overall lifespanI Project release windowsI Key external events


0100

200

300

400

Occ

urre

nces

(nor

mal

ized

)

1998 2000 2002 2004 2006 2008

Evolution -- Usability (R2 = 18%)

Year

Gno

me

1.2

Gno

me

1.4

Gno

me

2.0

Gno

me

2.2

Gno

me

2.4

Gno

me

2.6

Gno

me

2.8

Gno

me

2.10

Gno

me

2.10

Gno

me

2.10

Gno

me

2.12

Gno

me

2.16

Gno

me

2.20

Gno

me

2.24

Gno

me

1.0


Results

Correlation coefficient very low in both cases - no relationship

Analyzing peaks seems useful - external events are drivingfrequencies

“... I know this was discussed a few weeks ago ... could it beimplemented as an advanced option that has to be turned onand is off by default?”


Observations (2)

RQ2: Are qualities treated the same in different projects?

Examine normalized occurrences per week for extended signifiers

Quality Project Occurrences

Efficiency Evolution 0.012Nautilus 0.026

Usability Evolution 0.192Nautilus 0.285

Portability Evolution 0.010Nautilus 0.011


Threats to validity

1 How valid are word lists?

2 Domain-dependent quality models

3 Other sources of project discussions (IRC/F2F/Skype)

4 Open source projects may not generalize

5 No sense of ‘attitudes’ to requirements

6 Statistics


Future work: agenda

RQ3: Usefulness: Technique worked, but:

1 Leverage other machine learning techniques like multi-labellearning

2 Understand inter-project variations and usefulness of globaltaxonomies of quality

3 Use techniques to understand requirements evolution in softwareprojects

4 Tie experimental data to ethnographic data2

2Aranda, J. and Venolia, G.: The secret life of bugs: Going past the errors and omissions in software repositories. In

International Conference on Software Engineering, pp 298-308 (2009).


Happy Canada Day!


on the perception of software quality requirements during the project lifecycle

Technology