Chapter 2: Iterative, Evolutionary, and Agile

Choice of Software development Process Iterative because the entire project will be composed

of mini-projects and will iterate the same activities again and again (on different parts of the project) until completion.

Evolutionary (or incremental) because the software grows by increments (as opposed to the traditional Waterfall model of software development).

Agile because we will use a light approach to software development rather than a very rigid one (which may be needed for a safety-critical system for example)

This kind of approach seems better at treating software development as a problem solving activity; and the use of objects makes it amenable.

Unified Process A relatively popular iterative software development

process Iterative and evolutionary development uses

relatively early programming and testing of a partial system, in repeated cycles.

It typically also means that development starts before the exact software requirements have been specified in detail;

Feedback is used to clarify, correct and improve the evolving specification;

This is in complete contrast to what we usually mean by engineering! (Why? Why do you think it works better this way for software?)

What is the Unified Process (UP or RUP) and Why?

What: By the authors of the UML: Jacobson,

Booch and Rumbaugh in 1999 Not part of the UML !

Why: We need a process; It is popular for OO project; It is a good middle-of-the-road process;

Unified Process (cont.) The UP is very flexible and open and can

include other practices from other methods (e,g,Extreme Programming (XP) or Scrum). E.g. XP’s test-driven development, refactoring can fit

within a UP project; So can Scrum’s daily meeting. Being pragmatic in adapting a particular process to

your needs is an important skill : all projects are different.

The OO skills and techniques that we will see are not specific to the UP: They can be applied within any software

development process; The UP is just a framework for a project, it brings

some structure to what we do.

What is Iterative and Evolutionary Development? An iteration is a time-boxed (e.g. 3 weeks) mini-

project during which an increment of the software is developed and integrated into the current project.

Each iteration is made up of the following sequential activities: requirements analysis, design, implementation and testing.

The software grows by increments, and is refined (evolves) as we go along via feedback: Iterative and Evolutionary Development.

Be careful! -- an increment is not a prototype that sometimes work and sometimes doesn’t; it should represent a properly analyzed, designed, coded and tested chunk of the future software.

Fig 2.1 Iterative and Evolutionary Development

Requirements

Design

Implementation &Test & Integration

& More Design

Final Integration & System Test

Requirements

Design

3 weeks (for example)

The system grows incrementally.

Feedback from iteration N leads to refinement and adaptation of the requirements and design in iteration N+1.

Iterations are fixed in length, or timeboxed.

Time

Implementation &Test & Integration

& More Design

Final Integration & System Test

Example of three-week agile iteration (not always suitable)

The last iteration is debriefed and suggestion for process improvement are noted;

Feedback from previous demonstrations and subsequent requests for change are integrated in this iteration;

All the artefacts (use cases, UML diagrams, code …) from the previous iteration are updated and synchronized with each other;

Meeting clarifying the goals of the next iteration; Use cases may need to be refined; OOA is performed by the team in pairs with UML using

whiteboard and capturing outcomes using a digital camera; OOD is performed by the team again probably by pairs.

Previous UML artefacts are updated; Most of the remaining days are spent on implementation and

testing (with maybe re-design if necessary); The last day should be spent tidying up and documenting; (Other project management activities such as demonstrations,

quality evaluation, etc. have not been detailed here)

Agile iteration (cont.) Do not rush to code nor do a long drawn-out all-detailed

design phase; there should be team-dependent compromises.

The result of each iteration is an executable but incomplete system: It should not be a hack nor a prototype; For this approach to work we must not have to go back and

change previous design decisions or code (that’s a hack); Whatever we do within an increment must be thought as part

of the final system: it should be high quality, suitable for release, production-grade code and documentation;

Of course the above remarks are very important but: You will not always get it right: changing a previous iteration’s

code or design is inevitable and will happen all the time (the nature of the changes are important : are you re-designing all the time or just tinkering the code?); it is unrealistic to believe that the code of an iteration will never change during an entire project;

You must handle change during development;

Change is Inevitable because: What was done previously is not perfect (this is very common); The requirements were misunderstood (this is very common); The requirements change (this is very common);

Change is common. The waterfall-way of dealing with this problem is to spend enough time during analysis to ensure that the requirements are perfect: (generally impossible).

The UP and all other iterative approaches (Agile or not) try to embrace change: Each iteration looks at only one aspect of the project,

does so quickly, and generates an executable at the end; early feedback can be provided;

Entire detailed requirements do not need to be frozen: each iteration and subsequent evaluation-feedback clarifies the requirements that were looked at in previous iterations and also sheds light on the requirements of the future iterations.

The requirements are clarified as we go along: a problem solving process.

Iterative, Evolutionary and Agile process Of course, for this to work the resulting changes

must not be major all the time: We mustn’t make major blunders in our design; Or completely misunderstand a requirement; Or accept requirement changes (new feature please!)

from external sources (client, marketing department etc.) that require a re-design.

In general, changes get harder to accommodate as the project progresses.

We must not fall into a chaotic process but properly managed iterations during which release-grade artefacts are produced can successfully accommodate change. Up to a point …

Fig 2.2 Iterative feedback leads to an evolutionary system closer to the ‘real’ requirements

Early iterations are farther from the "true path" of the system. Via feedback and adaptation, the system converges towards the most appropriate requirements and design.

In late iterations, a significant change in requirements is rare, but can occur. Such late changes may give an organization a competitive business advantage.

one iteration of design, implement, integrate, and test

Benefits of iterative development Early progress that can be evaluated; Early feedback via user engagement gives

clarifications of requirements that will bring us closer to the ‘real’ requirements;

Complexity containment: we proceed by chunk; Early tackling of risks (technical,

requirements, objectives, usability etc.) Late total project failure less likely; Better productivity (especially when agile); Less defects; Possible process improvement within a single

project

Potential problems of iterative development

Going into a prototyping-hacking mode;

Poor documentation; Bad overall architectural design; Unsuitable for some software: safety-

critical (because safety properties must be proven true on the initial requirements);

Must strive for ‘good style’ all the time;

Each iteration should be time-boxed between 2 to 6 weeks:

Longer: rapid feedback and adaptation is lost, complexity of task is probably too great, falls into waterfall;

Shorter: quality and documentation will suffer; If the manager feels that a deadline will not be

met a recommended response is to de-scope : remove tasks or requirements from the iteration and schedule them to a future iteration, rather than slip the completion date (what do you think?)

The need for feedback and adaptation:

Feedback from the early demonstrations as client and developers try to understand each other to define and clarify the requirements as a team;

Feedback from testing implies that previous code needs to change;

Feedback from previous iterations to refine schedule and all other estimates (such as?);

Feedback from client and marketplace to change software emphasis (the marketplace does not stay still during software development …);

Overview of a simple example on 20 iterations (over a year) project:1. Before the first iteration, hold an initial

requirements workshop with client and senior developers. The UP Inception Phase:

o Define the objectives of the project;o Do high-level requirements analysis by identifying

use cases (e.g. 30) and features;o Identify non-functional requirements o Pick the 10% of the use cases that blend:

o Architectural Significance;o High marketing value;o High riskthis leads to say 3 use cases: UC2, UC11 and UC14;

o Do a detailed analysis of the functional and non-functional requirements for these three use cases;

o Review everything with everybody;

Ex. (cont.)2. Then the developers alone hold an iteration planning meeting

to define a subset of UC2, UC11 and UC14 for the first iteration to fit within a short (e.g. 4 weeks) timebox. Document the subset. Maybe tentatively schedule 2nd and 3rd iterations;

3. Iteration 1 (The UP Elaboration Phase):2. Perform OOA documenting the Domain Model using UML; review

by all developers (eg 3 days);3. Perform OOD documenting the Design Model using UML; in pairs,

review by all (eg. 2 days);4. Start programming, integrating and testing using previous UML

models as starting points; Try to update models; hold daily stand-up meetings (eg. 2 weeks);

5. One week before the end review and de-scope if necessary;6. On Tuesday of the last week: code freeze to provide increment;7. On Wednesday demo partial system to external stakeholders: get

and document feedback;8. 2nd requirements workshop: refine previous workshop document

and detail another important 10-15% of the use cases

Ex. (cont.)4. Iteration 2 : similar steps;5. Repeat for four iterations and five requirements

workshops so that at the end of iteration 4, perhaps 80% or 90% of the requirements have been written in detail but only 10% of the system has been implemented;

6. At the end of the Elaboration Phase, most of the requirements have been detailed by iterations and some of the system is available. All estimates should be reviewed

7. The UP Construction Phase can start: many more iterations, major requirements change is less likely during this period. Can be more ambitious on the amount of work to be performed. Keep learning and getting feedback;

8. In the UP the last phase is the Transition Phase: beta tests and deployment;

Fig 2.4: The Iterative Process

Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5

20%2%

requ

irem

ent

s

softwar

e

30%

5%

requ

irem

ent

s

softwar

e

50%

8%

90% 90%

20%10%

requirements workshops

Imagine this will ultimately be a 20-iteration project.

In evolutionary iterative development, the requirements evolve over a set of the early iterations, through a series of requirements workshops (for example). Perhaps after four iterations and workshops, 90% of the requirements are defined and refined. Nevertheless, only 10% of the software is built.

1 2 3 4 5 ... 20

week 1

M T W Th F

week 2

M T W Th F

week 3

M T W Th F

kickoff meeting clarifying iteration goals with the team. 1 hour

team agile modeling & design, UML whiteboard sketching.5 hours

start coding & testing

a 3-week iteration

de-scope iteration goals if too much work

final check-in and code-freeze for the iteration baseline

demo and 2-day requirements workshop

next iteration planning meeting;2 hours

Most OOA/D and applying UML during this period

Use-case modeling during the workshop

Risk-Driven and Client-Driven Iterative Planning Decide which use cases to schedule in the

elaboration phase, at the beginning of a project; select use cases according to: Risks : to drive down the highest risks early on; Client needs : to build visible features that the client

cares most about;How does this apply to games development?

Risk-driven iterative development includes the practice of architecture-centric iterative development; early iterations focus on building, testing and stabilizing the core architecture: why?

What are Agile Methods? Usually apply time-boxed iterative and

evolutionary development, employ adaptive planning, promote incremental delivery, and include other values and practices that encourage agility ( rapid and flexible response to change).

Examples of practices include: Common project workroom (from Scrum) Self-organizing teams (from Scrum) Programming in pairs (from XP) Test driven development (from XP)While we will not have the time to review every technique we will keep an ‘agile’ point-of-view in our process.

The Agile Alliance has issued a manifesto and a set of principles.

The Agile Manifesto:We are uncovering better ways of developing

software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

The Agile Principles:1. Our highest priority is to satisfy the customer through early

and continuous delivery of valuable software.2. Welcome changing requirements, even late in development.

Agile processes harness change for the customer's competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

The Agile Principles: (cont.)7. Working software is the primary measure of

progress.8. Agile processes promote sustainable

development.9. The sponsors, developers, and users should be

able to maintain a constant pace indefinitely.10. Continuous attention to technical excellence

and good design enhances agility.11. Simplicity−the art of maximizing the amount

of work not done−is essential.12. The best architectures, requirements, and

designs emerge from self-organizing teams.13. At regular intervals, the team reflects on how

to become more effective, then tunes and adjusts its behavior accordingly.

What is Agile Modelling? ‘The purpose of modelling is primarily to

understand, not to document.’ The waterfall calls for very detailed OOD documents that

are handed to a programmer; But modelling is still necessary; The models (UML diagrams) created are also very good

communication tools (face to face preferably); Don’t model everything: simple issues can be handled at

programming; Use the simplest tools as possible: prefer sketching UML

diagrams on whiteboards and capture the result with a digital camera; UML case tools can be used later to tidy things;

Do not model alone, or at least review by all; Create models in parallel (e.g. interaction diagrams and

class diagrams); All models are likely to be inaccurate in some respect; Best if whoever models is also responsible for coding;

Fig 2.5 Group-developed UML Sequence Diagram Sketch

What is an Agile UP? The UP can be used in an Agile

context: Stick to the spirit of an iterative and

evolutionary approach; Continuously engage users for

evaluation, feedback and requirements; Continuously verify quality; test early,

often and realistically; Practice change request management

and configuration management;

What are the UP Phases? 4 major phases:

1. Inception : approximate vision, business case, scope, vague estimates;

2. Elaboration : refined vision, use cases, iterative development of the core architecture, resolution of high risks, identification of most requirements and scope, more realistic estimates;

3. Construction : iterative implementation of the remaining lower risks aspects;

4. Transition : beta tests, deployment; This is not waterfall.

Fig 2.6 UP phases overview

inc. elaboration construction transition

iteration phase

development cycle

release

A stable executable subset of the final product. The end of each iteration is a minor release.

increment

The difference (delta) between the releases of 2 subsequent iterations.

final production release

At this point, the system is released for production use.

milestone

An iteration end-point when some significant decision or evaluation occurs.

What are the UP Disciplines?

The UP describes a set of on-going activities such as requirements analysis, design, testing, etc. These are spread over all phases (and

iterations); The relative effort across these

disciplines changes over time: e.g. early iterations tend to focus on requirements analysis rather than testing;

Fig 2.7 UP disciplines overview

Iterations

SampleUP Disciplines

Business Modeling

Requirements

Design

Implementation

Test

Deployment

Configuration & Change Management

Project Management

Environment

Focus of this book

Note that although an iteration includes work in most disciplines, the relative effort and emphasis change over time.

This example is suggestive, not literal.

A four-week iteration (for example). A mini-project that includes work in most disciplines, ending in a stable executable.

Sample

UP Disciplines

Business

Modeling

Requirements

Design

Implementation

...

incep-

tionelaboration construction

transi-

tion

...

Inception Elaboration Construction Transition

Iteration … Iteration … Iteration … Iteration …Iteration 1

OODCoding & Testing Integration

Use Case ASimplified

……

Use Case…

Use Case AFull…

Use Case BFull…

Use Case…

Use Case…

Use Case…

Use Case…

Use Case…

…

OOA

Adapting the UP Apart form the fundamental principles

(such as iterative and evolutionary, risk and client-driven development, realistic continuous verification of quality), the UP is very flexible.

We can write-up a Development Case document during inception to detail the choices of practices and artifacts that will be adopted for a project.

See Table 2.1, where ‘s’ stands for start and ‘r’ refine, for a partial sample Development Case.

Discipline

Practice ArtifactIterations ->

InceptionI1

ElaborationE1..En

ConstructionC1..Cn

TransitionT1..Tn

Business Modeling

Requirement workshop, mock UI

VisionDomain Model

s s

rr

Requirements

Requirement workshop, prototypes

Use Case model

s r

Design Agile modeling, testdriven development

Design ModelSW architecture document

ss

r

Implementation

Pair programming, testdriven development

Code s r r

Project management

Daily scrum meeting

… s r r r

…

You Know you did not Understand Iterative Development When …

You try to define all the requirements details before you start (or the entire detailed design;

You spend weeks doing UML modelling before programming;

You think that inception = requirements analysis, elaboration = design and construction = coding;

Your iterations are months long rather than weeks; Your plans are fuzzy, overly detailed, document led and

rigorously enforced; People work in isolation, handing one document after

another to the next person (e.g. requirements specification from analyst to a designer, design document to programmer, code to tester);

Conclusion We will apply the UP, an iterative and

evolutionary process, within an agile context.

Resources: www.agilealliance.com (the Agile Alliance) www.agilemodeling.com (Scott Ambler) Craig Larman “Agile & Iterative Development: A

Manager’s Guide”, Addison Wesley, 2004:

Questions?