What We Think We Know About Software Development - and Why We Believe it’s True

Steve Crouch, SSIs.crouch@software.ac.uk

with thanks to Greg Wilson

Software Carpentry Workshop, Newcastle14/05/2012

Why?• Researchers are expected to just

know/find out about programming on demand

• American car manufacturers– Compete on e.g. cost– But not quality– Enter Volkswagen, BMW,

Mercedes!• Building things right allows you to

build things faster• Condition of publication– Data archival of results– What about software that

generates the data

Image courtesy of M 93

Once Upon a Time…

7 Years War (1754-63)

Britain loses 1512 sailors to enemy action…and almost 100,000 to scurvy

A Twist of Irony…• True irony, not Alanis

Morrisette irony• James Lind (1716-94)• 1747: (possibly) first ever

controlled medical experiment

Cider Sea water

Sulphuric acid Oranges

Vinegar Barley water

• Ignored – not a proper English gentleman!• Listened – repeated in 1794 by a…

Courtesy of the National Library ofMedicine

A Twist of Irony…• True irony, not Alanis

Morrisette irony• James Lind (1716-94)• 1747: (possibly) first ever

controlled medical experiment

Cider Sea water

Sulphuric acid Oranges

Vinegar Barley water

• Ignored – not a proper English gentleman!• Listened – repeated in 1794 by a…

Courtesy of the National Library ofMedicine

Towards Modern Medicine• Medical profession eventually

realised controlled studies were the right way to go

• David Sackett, 1992: ‘evidence-based medicine’– Randomised double-blind test is gold

standard

• Cochrane Collaboration– Archives results from hundreds of

medical studies– http://www.cochrane.org/

Image courtesy of Lab Science Career

On to Software Engineering…

• Martin Fowler (IEEE Software, July/Aug 2009)– Wrote the book on refactoring

• “[Using domain specific language] leads to two primary benefits. The first, and simplest, is improved programmer productivity... The second... is... communication with domain experts.”

• Interesting! Image courtesy of adewale_oshineye

‘Just One More Thing…’• Perhaps seems like a good idea• Look at the paper more closely…– Two substantive claims– …without a single citation!

• Could be correct, but where’s the data?

• A Scottish verdict!• Even the best of us aren’t doing

this correctly

• “Many software researchers advocate rather than investigate”– Robert L. Glass (2002)

Image courtesy of whatleydude

Image courtesy of futureatlas.com

Take Note… Things are Starting to Change

• Growing emphasis on empirical studies in s/w research since mid-1990s– Papers describing new tools or

practices routinely include results from some kind of field study • ICSE 2009

• Almost always must include some field testing results in top-tier journal/conference papers now – Progress!

Image courtesy of Hacklock

An Interesting Result…Aranda & Easterbrook (2005): “Anchoring and Adjustment in

Software Estimation”

“How long do you think it will take to make a change to this program?”

3 groups given two page spec, one variation per group embedded in text…

Control Group: “I’d like to give an estimate for this project myself, but I admit I have no experience estimating. We’ll wait for your calculations for an estimate.”

Group A: “I admit I have no experience with software projects, but I guess this will take about 2 months to finish.”

Group B: “… I guess this will take about 20 months to finish.”

Groups don’t know about this variation…

What Happened?

• This is an anchoring effect, well known in psychology• The anchor mattered more than anything else:– Experience in domain, software engineering– Tools used– Formality of estimation

• Q: Are agile projects similarly afflicted, just on a shorter and more rapid cycle?

Group A (lowball) 5.1 months

Control Group 7.8 months

Group B (highball) 15.4 months

Frequently Misquoted

Sackman, Erikson and Grant (1968): “Exploratory experimental studies comparing online and offline

programming performance”

“The best programmers are up to 28 times more productive than the worst.”

• Dozens of books quote this – ‘all you need is a few really, really good people’– More productive than… others!– We put ourselves at ‘good’ end of bell curve!

Pick it Apart

• Done in 1968!!• Study designed to compare batch vs

interactive, not measure productivity• How was productivity measured? No

metric• Best vs worst?• Twelve programmers for an afternoon– Next “major” study was 54

programmers…– … for up to an hour

• Very bad use of statistics!

Image courtesy of zolierdos

So What Do We Know?

• Lutz Prechelt• This guy knows stuff – via studies– Productivity variations between

programmers– Effects of language– Effects of web programming

frameworks

• Productivity and reliability depend on length of program’s text, independent of language abstraction level• e.g. Perl, Ruby, Java, Assembler - ~the same LOC/hour• Therefore: go high-level (trade-off: performance!)

A Classic Result…

• Boehm et al (1975): “Some Experience with Automated Aids to the Design of Large-Scale Reliable Software”– Plus many, many more since

1. Most errors are introduced during requirements analysisand design

2. The later they are removed, themore expensive it is to take them out (1/10/100?)

Operational: 1989 - 200322,000 lines of code122 software errors79% because of poor understanding of requirements

Greatest Hits Volume 1

• For every 20% increase in problem complexity, there is a 100% increase in solution complexity (Woodfield, 1979)

• The two biggest causes of project failure are poor estimation and unstable requirements (van Genuchten, 1991 – and many others)

• Development teams do not benefit from existing experience, and they repeat mistakes over and over again (Brossler, 1999)

Greatest Hits Volume 2• Rigorous inspections can remove

60-90% of errors before first test is run (Fagan, 1975)

• The first review & hour matter most (Cohen, 2006)

• Physical distance doesn’t affect post-release fault rates. Distance in organisational chart does Nagappan et al (2007) & Bird et al (2009)

• Shouldn’t our development practices be built around these facts?

Image courtesy of schoschie

Here’s Another

• Errors tend to cluster– Half the errors are found in 15% of the modules

(Davis, 1995 quoting Endres 1975)– About 80% of the defects come from 20% of the

modules, and about half the modules are error free (Boehm and Basili 2001)

• Known since 1975!

When you identify more errors than expected in some program module, keep looking!

Learn to Read• How did you learn to program in any

language?• But…

– You don’t just go out and write War and Peace, you’d read other skilled writers first

– You learn to read a foreign language before you learn to write it

• Err… what about programming languages?

• Teach maintenance first (Harlan Mills, 1990)

The best way to prepare [to be a programmer] is to write programs and to study great programs that other people have written– Susan Lammers, 1986, quoting a younger Bill Gates

Image courtesy of dwyman

And Finally…

• Get into good habits early• It’s easy to cling to bad ones

• http://www.python.org/doc/humor/#bad-habits