1

Fast & FlexibleSoftware Development

Michael A. Cusumano

MIT Sloan School of Managementcusumano@mit.edu

© 2006

2

3

Different Cultural Orientations

EUROPE: Software as a ScienceScience–Formal Methods, Object-Oriented Design

JAPAN: Software as ProductionProduction–Software Factories, Zero-Defects

INDIA: Software as a ServiceService–Infosys, Tata, Wipro, Satym, Cognizant, Patni

The USA: Software as a BusinessBusiness–Windows, Office, Navigator, $$$$

4

Problems in Software Development

• Similar problems recurring since the 1960s

• 1969 NATO Report on Software Engineering:

Documented problems in – requirements, design vs. coding separation– estimates, monitoring progress, communication– productivity (26:1), metrics, reliability (bugs) – hardware dependencies, reuse – maintenance costs

• Sound familiar??

5

Proposed Solutions• Many attempts

– IBM-style software engineering (1960s, 1970s)– Japanese “software factories” (1970s, 1980s – stable

teams, standard process & tools, reuse)– SEI Capabilities Maturity Model (1980s to present)– “Iterative” and “Agile” methods (US, Europe)

• No one process perfect for all projects– Variations: business models, customer requirements,

application domain, competition, pace of change, local culture??, etc.

• Common Theme: How balance quality, features & design flexibility, with cost & speed

6

Different Process Philosophies• Waterfall-style (sequential, “Stage-gate”)

versus

• Iterative-style (agile, incremental)– Spiral– Rapid Prototyping– Synch-and-Stabilize (Microsoft, PC makers)– IBM Rational’s Unified Process & Toolkit– HP’s Evo Process (short cycles of mini-waterfalls)– Extreme Programming (XP), SCRUM, AGILE– Many other variations at companies

7

Traditional Waterfall Model(One development cycle)

Requirements Functional design

Detailed module design

Module construction Module construction Module construction

Integration/system test

Module rework (debug)

Re-test, debugging

Product release

8

Frequent Waterfall Process Result

Requirements Functional design

Detailed module design

Module construction

Module construction Module construction

Integration/system test

If modules change a lot, integration fails and you can experience an infinite defect loop.

9

Reality: Spectrum of ApproachesProcess Choices for Different Projects

User FeedbackDuring Project

Uncertainty in Requirements

Early/OftenHigh

Late/Occasional Low

# of Cycles, Releases1 Many

Traditional Waterfall, Older “Factory-like”Approaches

RapidPrototyping

XP

Agile or Iterative Methods

Incremental

Adapted from Bill Crandall (HP)

10

SYNCH-&-STABILIZE WATERFALL (or other Iterative styles) Spec, Coding, Testing in Parallel Separate Phases in Waterfall Sequence Vision Statement & Evolving Spec “Complete” Spec Document (specification = output, not input) & Detailed Design Before Coding Prioritized Features Built in 3 or 4 All Pieces of a Product Built Milestones Simultaneously Frequent Synchs (Daily Builds) & One “Late & Large” Integration & Intermediate Stabilizations (Milestones) Test Phase at Project End “Fixed” Ship Dates & Multiple Attempt to Achieve Feature & Release Cycles Product “Perfection” Customer Feedback During Development Feedback as Input for Future Projects

Iterative Style(“Synch & Stabilize”) Waterfall-Style

11

International Comparisons (2003 IEEE Software, “Software Dev. Worldwide”)

• Survey: Completed in 2002-2003, with Alan MacCormack (HBS), Chris Kemerer (Pittsburgh), and Bill Crandall (HP)

• Objective: Determine usage of iterative (Synch-&-Stabilize) versus Waterfall-ish techniques, with performance comparisons

– 118 projects plus 30 from HP-Agilent for pilot survey

• Participants

– India: Motorola MEI, Infosys, Tata, Patni

– Japan: Hitachi, NEC, IBM Japan, NTT Data, SRA, Matsushita, Omron, Fuji Xerox, Olympus

– US: IBM, HP, Agilent, Microsoft, Siebel, AT&T, Fidelity, Merrill Lynch, Lockheed Martin, TRW, Micron Tech

– Europe: Siemens, Nokia, Business Objects

12

“Conventional” Good PracticesIndia Japan USA Europe etc Total

Number of Projects 24 27 31 22 104

Architectural Specs % 83% 70% 55% 73% 69%

Functional Specs % 96% 93% 74% 82% 86%

Detailed Design % 100% 85% 32% 68% 69%

Code Generators -- Yes 63% 41% 52% 55% 52%

Design Reviews -- Yes 100% 100% 77% 77% 88%

Code Reviews -- Yes 96% 74% 71% 82% 80%

13

“Newer” Iterative PracticesIndia Japan USA Europe etc Total

No. of Projects 24 27 31 22 104

Subcycles -- Yes 79% 44% 55% 86% 64%

Beta tests -- Yes 67% 67% 77% 82% 73%

Pair Testing -- Yes 54% 44% 35% 32% 41%

Pair Programmer -- Yes 58% 22% 36% 27% 35%

Daily Builds at project start 17% 22% 36% 9% 22%

In the middle 13% 26% 29% 27% 24%

At the end 29% 37% 36% 41% 36%

Regression test each build 92% 96% 71% 77% 84%

14

“Crude” Output ComparisonsIndia Japan USA Europe

etc.TOTAL

Projects 24 27 31 22 104

LOC/

Month

median 209 469

cf. 389 in 1990

270

cf. 245 in 1990

436 374

Bugs/ 1000 LOC

median .263 .020

cf. .20 in 1990

.400

cf. .80 in 1990

.225 .150

15

Observations from Global Survey• Most projects (64%) not pure waterfall; 36% were!• Mix of “conventional” and “iterative” common -- use of

functional specs, design & code reviews, but with subcycles, regression tests on frequent builds

• Customer-reported defects improved -- over past decade in US and Japan; LOC “productivity” may have improved a little

• Japanese still report best quality & productivity -- but what does this mean? Preoccupation with “zero defects”? Need more lines of code to write same functionality per day as US & Indian programmers?

• Indian projects strong in process and quality -- but not as strong as their dominance of CMM Level 5 suggests??

16

Hewlett Packard Pilot Study (2003 IEEE Software, “Tradeoffs” article)

• Managers – When use iterative or waterfall?– Survey: 35 responses, 29 projects with complete data– Median – 170K LOC, with 70K new code; 9-person team, 14

month projects– 59% applications, 38% systems, 28% embedded

• 74% of variation in defects explained by early prototypes, design reviews, and integration/regression testing on builds– Median project: 40% of functionality complete when first

prototype released and 35.6 defects per million (.04/1000) LOC, reported by customers in 12 months after release, and 18 LOC per person day (360/month)

17

Multivariate Regression AnalysisSome striking results, compared to median:• Releasing prototype earlier with 20% of functionality

27% reduction in defect rate• Integration/regression testing at each code check-in

36% reduction in defect rate • Design reviews • 55% reduction in defect rate

• Releasing prototype with 20% of functionality 35% rise in LOC output/programmer

• Daily builds • 93% rise in LOC output/programmer

18

Observations from HP Survey • Best “nominal” quality from traditional “waterfall” (fewer

cycles & late changes = less bugs, of course!!)

• Best balance of quality, flexibility, cost & speed from combining conventional & iterative practices

• BUT: Differences in quality between waterfall & iterative disappear if use a bundle of techniques:– Short development subcycles (subprojects/milestones)– Early prototypes to get customer feedback– Frequent builds to incorporate feedback, changes– Design/code reviews (check quality continuously)– Regression tests on each build (check for errors, late changes,

integration problems)

19

Where Iterative May Work Better• Fast-paced product markets where technologies,

competition, and product features may have to change a lot during a project.

• Product or custom projects where customers place very high emphasis on leading-edge features.

• Custom systems where customers place high emphasis on responsiveness to their input during a project.

• Product or custom projects that require experimentation as in lots of short design, build, and test cycles.

• Other?

20

Where Waterfall May Work Better• Extremely high-reliability systems (product or custom

projects), where functions are very well understood and no changes in requirements during a project are desired

• Embedded products with hardware constraints that cannot be easily changed

• Contract software where client requires a detailed proposal upfront and not possible to limit scope/phases

• Complex projects (product or custom) where parts of design & coding are outsourced, off-shored, or done in multiple sites, AND there are weak mechanisms to synchronize and manage distributed teams

21

• High-Level Process

• Innovation & Design

• Architecture

• Team Management

• Project Management

• Testing & QA

Some Comments on Good Practices

22

High-Level Process Strategy

• “You can’t do anything that’s complex unless you have structure.”

• For creative people, that structure should be as subtle as possible.

23

Dave Maritz Test Mgr, Windows 95

Everything that I do here I learned in the military.... You can't do anything that's complex unless you have structure .... And what you have to do is make that structure as unseen as possible and build up this image for all these prima donnas to think that they can do what they like. Who cares if a guy walks around without shoes all day? Who cares if the guy has got his teddy bear in his office? I don't care.... [But if] somebody hasn't checked in his code by five o'clock, then that guy knows that I am going to get into his office.”

From Microsoft Secrets

24

High-Level Process cont’d• Structure:

– Process must fit the product & market/customer

• Nature of the Product or Service– Mission critical vs. not; new vs. derivative– Infrastructure vs. applications, embedded, etc. – Packaged vs. custom, or multi-system solution

• Nature of the Market or Customer:– Speed & innovation for leading-edge markets– Quality & support for enterprise & mass mkts

25

Innovation & Design Strategy• Being “slightly out of control” can stimulate

innovative thinking and creativity.• But too few controls chaos or “rocket science”

• Iterative a middle approach: Vision statements, outlines of functional specs, prototypes, early beta releases, multi-version release mentality, etc.

• Late design changes can be good: help improve the product -- respond fast to user feedback, competitors’ moves, or unforeseen changes in the market & technology.

26

Architecture Strategy• Definition:

– How to divide a product into subsystems and modules/components, and with what interfaces

• Strategy: – degree of “modular” vs. “integral”– degree of “open” vs. “closed” (proprietary)– consider/beware of “Open but not Open”!

• Modular & “horizontal” (feature-oriented)– de-couples small development teams– facilitates adding or cutting functionality– facilitates reuse of components

27

Architecture Strategy

Tradeoff:

• Investment in architecture for the future versus new features for the present

• Try to design architectures that will last. – Rush-jobs “spaghetti” & hard to modify

• Or, plan to evolve architectures incrementally

28

Bob Lisbonne, Netscape VP Client Development

When our teams grew beyond a certain point, they began to resemble a 200-person three-legged race … That’s why the componentization or the modularization of the product is so key, so that ultimately we can get back to lots of small teams each doing their own thing… and not getting caught up in one another’s efforts.

From Competing on Internet Time

29

Team Management Strategy• Small teams of great people work best

• Large teams can work like small teams (within limits – not Longhorn!)

• What you need:– One strong person in charge!– Map project architecture to product architecture

feature & subsystem teams– Keep feature teams small 3-8 people. – Overlap functional responsibilities (vision/scope,

requirements, QA)– Focus everyone on shipping the product!

30

Chris Peters, VP Microsoft Office

Everybody in a business unit has exactly the same job description and that is to ship products. Your job is not to write code, your job is not to test, your job is not to write specs. Your job is to ship products.... When you wake up in the morning and you come to work, you say what is the focus? Are we trying to ship? Are we trying to write code? The answer is we are trying to ship.... You're trying not to write code. If we could make all this money by not writing code, we'd do it.

From Microsoft Secrets

31

Project and Feature Team Organization

Note: Bold type indicates project leaders. Ggroup

Product Unit Manager

Product Planners

Program Manager Developer Team Lead Developer ---- Developer ---- Developer ---- Developer ----

Tester Team Lead ----- Tester ----- Tester ----- Tester ----- Tester

Group Program Manager Development Manager Developer Team Lead Developer ---- Developer ---- Developer ---- Developer ----

Test Manager Tester Team Lead ----- Tester ----- Tester ----- Tester ----- Tester

Program Manager Developer Team Lead Developer ---- Developer ---- Developer ---- Developer ----

Tester Team Lead ----- Tester ----- Tester ----- Tester ----- Tester

User Education Staff

Customer Support Product Specialists

Microsoft Project Team Structure

32

Project Management Strategy

• Avoid sequential “waterfall” schedules (usually).

• Divide long projects into multiple sub-projects or milestones, of a few weeks or months duration.

• Evolve requirements incrementally: do spec design, development & testing concurrently.

• Impose a few rigid rules to force frequent synchronizations and periodic stabilizations.

Synchronize-and-Stabilize!

33

Project Management cont’d

• Prioritize Rigorously: most important features first

• Schedule “backwards” – people & time

• Let engineers schedule their own tasks

• Managers keep historical data on estimates

• Prototypes and early beta releases = rapid feedback on designs and quality

• Use historical data to schedule buffer time (for projects, not individuals) for changes & unknowns

34

Synchronize-&-Stabilize Product Vision/Architecture Design Functional Specification Milestone 1 Milestone 2 Milestone 3 Design Design Design Code Code Code Usability test Usability test Usability test Test Test Test Daily builds Daily builds Daily builds Test Test Test Debug Debug Debug Integrate Integrate Integrate Stabilize Stabilize Stabilize Buffer time Buffer time Buffer time Alpha release Beta release Feature complete Beta release Visual freeze CODE COMPLETE Final test Final debug Stabilize Final release

Synch-and-Stabilize Process

35

Synch-Up and Debug Daily• Microsoft Secrets: Few rules, but “military-

like” discipline to force coordination & communication– Developers can check in when they want – Each project must build daily– Check-in at set times; can’t leave until build is OK– If you break the build, you must fix your code now! – Penalties for those who break the build

• Minimize build overhead through build team and automated toolsObjectives:– Synching-up: 5-20 minutes; Quick test: 30

minutes – Total Check-in: 5-60 minutes

36

Testing and QA Strategy

• Try to build-in quality continuously– design and code reviews– gates or check points– continuous customer feedback, etc.

• But continually integrate and test– Frequent builds (daily, weekly)– Especially check late design changes

• Test what?– unit/feature testing, of course– system integration testing -- from early on!

37

Testing and QA cont’d

• Automate to test & retest design changes quickly and frequently.

• But automation never eliminates the need for people. Someone has to write and rewrite (update) the automated tests.

• Also need human testers to probe real user behavior

38

Testing and QA cont’d

• Post-mortems: what went well, what went poorly, and what the team should do next time.

• “Eat your own dog food”: first-hand feedback on products as quickly as possible.

• A few quantifiable metrics: to control and improve quality as well as monitor key product and process characteristics.

• Early beta releases: provide feedback on design as well as quality.

39

Concluding Comments• No one “best” software development process

– But “waterfall” less responsive to change– But using a “bundle” of practices eliminates

differences in quality between waterfall and iterative!

• Process should depend on business, context, and strategy– Type of software, customer requirements, team

experience and culture, contract needs, etc. – Product vs. custom, mass-market vs. niche, individual

vs. enterprise, leading-edge vs. follower, etc.

40

Main References• The Business of Software by M. Cusumano (Free Press/Simon & Schuster,

2004)

• Microsoft Secrets by M. Cusumano and R. Selby (Free Press/Simon & Schuster, 1995 and 1998)

• Competing on Internet Time by M. Cusumano and D. Yoffie (Free Press/Simon & Schuster, 1998)

• Michael Cusumano, Alan MacCormack, Chris Kemerer, and Bill Crandall, “Software Development Worldwide: The State of the Practice”, IEEE Software, November-December 2003. (International Comparisons)

• Alan MacCormack, Chris Kemerer, Michael Cusumano, and Bill Crandall, “Trade-offs between Productivity and Quality in Selecting Software Development Practices”, IEEE Software, September-October 2003. (HP Survey)