may, 2004william (bill) myers1 agile methodologies and chaos revisited william (bill) myers cinergy...

May, 2004 William (Bill) Myers 1

Agile Methodologiesand

Chaos RevisitedWilliam (Bill) Myers

Cinergy Corporation

Information Builders Summit 2004 User Conference

May 2004

* The statements and opinions herein are those of the individual author. They are provided for informational purposes only and should not be relied upon for decision making.

May, 2004 William (Bill) Myers

Introduction• Audience & Flow

• Agile Methodologies • Philosophy & Methods

– Management & Coding

• General Agile reporting• Questions

– Follow-up questions at end

• Appendix • Links • Additional Details / starts at References slide• Additional points

• Supplemental Handout


What is Agility

• Ability to adapt • to Changing circumstances• to Changing requirements

• Impact• Improves business climate• Provides competitive edge• Quick gratification

» Customers & Developers» Faster Return on Investment


Point of View - Concerning Risk

• Traditional Methods• Risks for IT

– Errors– Delivery on budget, on time, on scope

• Business risks generally ignored

• Agile Methods• Business risks

– Competitive timeliness– Effectiveness / the right thing

• IT risks– Errors– Budget, time, scope


Agile Methodologies / Perspective

• How to Decide

• Relationships & Responsibilities

• 80 / 20 rule• Feature driven• Customer stories• Baseline high level• Emerge/Converge

• Reduce reporting risks

Note: Crystal Methodology IS NOT RELATED to the reporting product


What is Agile Development?• Set of policies and conventions

• Controls risk• Puts project into safety zone

• Agile Methodologies• Changing environments • Philosophies

– Products, Personal relationships, Timing, Documentation

• Methods– Approach & Principles– Project management– Programming – low level practices

• Traditional Methodologies• Rigid, known environments• Static processes• Structured Methods

– Waterfall project management & clones


Agile Philosophieshttp://www.agilealliance.org

• Agile Manifesto– Individuals and interactions

– over processes and tools

– Working software– over comprehensive documentation

– Customer collaboration– over contract negotiation

– Responding to change – over following a plan

Quote from consortium: “While there is value in the items on the right (sub-topics above), we value the items on the left (major topics) more”.


General Agile Principles• Deliver something useful to customer

• Nothing counts until you deliver software

• Deliver frequently• Four to eight weeks

• Encourage collaboration• “Active customer involvement is imperative!” – find a champion• Daily meetings – very short

• Provide technical excellence• Rely on the skills of your people• Do the simplest thing possible

• Balance flexibility and structure• Self-correcting teams

• Don’t let people suffer• Install when business needs it & technically feasible


Agile Manifesto - Highlights

• Nothing counts until you deliver software• Deliver every four to eight weeks• Direct contact - customers & developers• Daily contact and communication• Let champions drive projects • Maintain a constant pace• Agile processes enhance quality• Teams self-correct• Never forget the goal - software


Adaptability ScaleRemember Sarbanes-Oxley


Project Management Styles• Command & Control

• Defined process• Plan what you expect – make predictions• Enforce the predictions• “Change Control”

– to manage change

• Leadership – Collaboration• Empirical process• Start with a plan• Change plan frequently• Inspection & Adaptation

– To control change


Waterfall Methodsand clones

• Process• Sequential• Analysis, design, construct, test, implement• Rigid• Paper intensive• Good with 1960’s mainframes with cards

• When Useful• Known processes

– Hardware & vendor software upgrades

• Little creativity required


Many Agile Methods • What methodology to use ?

• Crystal & Crystal Light ( helps with decision )

• Project management• Scrum ( broadest, variable end-point )• DSDM Model ( construction oriented )• Lean Development (domain, 80/20 approach )• Feature-Driven Development - FDD (most structured)• Adaptive• Extreme Programming – XP

• Programming oriented• Extreme Programming - XP

» Customer Stories, programs, working conditions

• Agile reporting ( core & option )


Agile Methodologies• Agile Software Development Ecosystems

• Reference book & web site• by Jim Highsmith• http://www.jimhighsmith.com

• Tom DeMarco: • Agility: 1• everything else: 0

• Numerous books• Highlights only


Observations - Agile Conference• Easy to go wrong

• Mis-interpretation, mis-implementation, missed point

• No silver bullet• One size does not fit all

• Beware too much documentation• Offload legal requirements from developers

• Always “on time, on scope, on budget”• Leads to mediocrity ( no risks taken )

• Cultural change• from - military Command & Control• to – cooperative Leadership & Collaboration

• Productivity leaps• 77 function points / month when industry average 2 pts / mo • WebFocus fits agile development very well


Reasons for Failure• One Size Methodology Fits All• Intolerant Methodology• Embellished

• Do frivolous tasks• Ignore necessary tasks

• Heavy• Untried• Tried Once

• Limited applicability• Champion

• Failure to Tune• Distractions

• Dates can’t be met


Crystal Methodology – Theory

( NOT RELATED to Crystal Reporting )

• Self-adapting "shrink-to-fit," software development methodologies

• Different set of policies & conventions for each project• Projects are sensitive to people issues • Crystal is people-centric

– other methodologies may be: » process-centric» architecture-centric» tool-centric

• Keys• Communication intensive

– Masses of paper do not improve communication• Experiment based • Requires feedback loops

• Theory of Methodologies


Crystal – Communication

• Effectiveness of different methods

• Courtesy of Alistair Cockburn • http://alistair.cockburn.us/crystal/articles/cpanfocisd/characterizingpeopleasnonlinear.html

People are Non-Linear

First-Order Components in Software Development


Crystal - Model• Model

• Risk to end-user, Size, Complexity– Tables present questions to ask

» To establish processes & controls» When to change

• Weight is costly– Heavy methods for large teams– More ceremony for critical deliverables– Avoid frivolous tasks– Efficiency is expendable

» Non-bottleneck activities & Rarely used items

• Crystal Methodologies• Reference books & web site• Alistair Cockburn – Author• http://alistair.cockburn.us/crystal


SCRUM• Principles

• Do the right thing (from customer’s view)• Deal with reality, not artifacts or fixed plans• Short iterations

– Usually four weeks– Be prepared to implement at the end of an iteration

• Allow changes• Keep everything visible

– Concentrate on time remaining - not time spent

• Highlights and pinpoints responsibilities• Self organization

– Creates sense of obligation and teamwork– Extensive collaboration– Changes role of Project Leader

» Certification available


SCRUM• Broad application

• Iterative, lightweight project management method• Software AND non-software projects• Based on manufacturing Process Control theory

• Adaptable – Can be implemented

• Beginning of a project• Middle of a project• Product development effort that is in trouble

• More information• Ken Schwaber and Mike Beedle • http://www.controlchaos.com/


SCRUM – Concepts

Courtesy of Ken Schwaberhttp://www.controlchaos.com/Scrumo.htm


SCRUM• Full visibility

• Priority list• Items in development cycle ( sprint )• Short, daily meetings ( 15 minutes )• Work hours remaining in sprint• Assignments, who & what

• Responsibility & Authority• Anyone can add to priority list• Business manages priorities• Developers determine what can be done• Project Leader removes impediments

• Removes developers from politics


SCRUM - Impact• Backlog

• Scope Creep – no such thing• No request ever lost

» It may never get a high enough priority

• Business owner only one who can remove it

• Return on Investment• Install option available

– Software completed to date available / immediate return

• Option to stop– Save development costs

• Perfect for fixed price projects– Reorder deliverables list – no cost– Substitute deliverables with equal effort – no cost– “Stop Development” clause easy to apply– Big competitive advantage


Scrum - Definitions• Product Backlog

• Prioritized list of items» Business Product Owner – sets priorities» List of Functional requirements,

Nonfunctional requirements, Issues» Contains preliminary estimate» Changes constantly» Items in a sprint: locked down

• Time-box• Period of time that can not be exceeded

• Sprint• Time-box of 30 days• Sprint backlog defines delivery list

» Product item» Internal support item (i.e. database, system doc)» Work remaining & burndown graph


Scrum - Players• Scrum

• Product Owner» Adds items to product backlog list» Set priorities

• Team» Determines sprint list» Develops software

• Stakeholder» Funded project, will use it, affected by it» Requests enhancements

• ScrumMaster» Responsible for Scrum process» Removes impediments

• Other interested parties• IT Management

» Manpower allocation» Budgets & Billing

• Senior Business Management» Best use of corporate resources


Scrum - Milestones

• Sprint Planning meeting ( two 4 hour segments )» Product owner & Team: identifies backlog items to be

delivered in 30 day sprint / Team commits» Team plans details in Sprint Backlog

• Daily Scrum Meeting ( 15 minutes )» Team: Review prior day, identify impediments, plan day

• Sprint Review Meeting ( 4 hours )» Team: demonstrates completed functionality

- to Product Owner and other interested parties

• Sprint Retrospective meeting ( 3 hours )» Team: how to improve next sprint or

make more enjoyable

Sprint PlanningMeeting

( 8 Hours )

Sprint( 30 cal days )

Sprint ReviewMeeting

( 4 hours )Daily Scrum

Meeting( 15 minutes )

SprintRetrospective

Meeting( 3 hours )


SCRUM: database - goals• Software development

• Collect new requests• Manage product backlog• Manage Scrum process• Reduce administrative load on developers

» Reduce or eliminate most status reporting needs

• Support Management• Provide data for manpower planning• Provide status to everyone

» Priority in queue» Currently being worked on» Remaining effort

• Provide historical information» Time tracking mechanism for billing» Answer “What did you do for us last year”


SCRUM – Process FlowSCRUM

Bu

sine

ssP

rod

uct

Ow

ner

De

velo

pm

en

tT

ea

mA

ppl

icat

ion

Re

po

rts

Use

r

Request newfeature

Accept orRejectrequest

ProcessResponse

Request

via Web Page

e-Mailresult

Backlog

Result

Makehigh levelestimate

PrioritizeBacklog

e-Mail notification

Estimate

Sprint List

Determine whatcan be done

by priority(Team Commits)

Commitsbusinessresources

PlanningMeeting

Hours Workremaining

Chargeback

Daily SCRUM Meeting:

1. Completed yesterday2. To complete today3 Impediments

Log time forbilling only

DemonstrateResults

Daily

Putinto

Production?

Put intoProduction

Yes

Status Change

ProjectedManpower

Needs

StatusReports

InvoicedHours

CurrentAssignments

Work inProcess

DeliveredProducts

Priorities

RetrospectiveMeeting

REPORTS:

Timely - reports always show current status, but can show historical strips Agile - reports are self-building, self-summarizing, include many filters Visible - only a confidential project would be hidden

Backlog Planning Sprint Demonstration

Accepted

William W Myers - June 2004


SCRUM – Database ModelProjects

Project_ID

Comments

Comment_ID

RequestFromUser

UserReq_ID

Features

Feature_ID

Assignments

Project_IDPerson_IDFeature_ID

ProjectTeam

Project_IDPerson_ID

People

Person_ID

ProjectManagement

ProjMgt_ID

HoursCharged

Charge_ID_IDPerson_IDSource

ModuleSprint

Module_IDSprint_ID

ProductionModule

Module_ID

Sprints

Sprint_ID

SprintFeatures

Sprint_IDFeature_ID

Sprint_to_Feature

Comment_IDFeature_ID

ProjectType

ProjectType

SCRUMDatabase

Model

William W Myers - June 2004


SCRUM: demo app – Priority Page


DSDM Model

• Principles• Iterative & incremental development• Changes during development are reversible• Requirements baselined at high level

– Minimum details to continue

• Testing throughout lifecycle– Developers AND business users

• More Information• Developed in UK• Grew out of RAD approach• http://www.dsdm.org/


Lean Development Model• Most strategic oriented of Agile methods• Principles

• Domain instead of point solutions• 80% today better than 100% tomorrow• Everything is changeable• Minimal is essential• Needs determine the technology• Best value for money

• More information• http://www.poppendieck.com/• http://www.leanconstruction.org/


Adaptive Software Model• Software development

• Complex adaptive system• Like stock market or flock of birds• Circumstances generate emergent capabilities

» Total skills exceeds sum of parts » Only emergent capabilities tames complexity

• Software organizations:Must create the proper circumstancesfor emergenceof superior capabilities


Adaptive Software Model

• Principles– Leadership – Collaboration

– Speculate, collaborate, learn– Drop militaristic Command and Control (stifles emergence)

– Cycle through phases many times, converging– Tasks not listed in plan– Components at end of plan are listed– Mission directed, feature driven, iterative

time boxed, risk driven, change tolerant


Feature-Driven Development (FDD)

• Process– One-time

– Quick, high value, low dollar

• Develop overall domain model• Build feature list• Plan by feature

– Iterative• Design by feature• Two week cycles building components

• More information• http://www.featuredrivendevelopment.com/• http://www.nebulon.com/fdd/


Extreme Programming - XP• Project management

• Collect customer stories• Estimate & prioritize• Small iterative development cycles

• Development• Test-first development

» Write test cases before coding• Collective code ownership• Refactoring• Pair programming• Daily standup meetings

• Many processes & rules• More information

• http://www.extremeprogramming.org• http://www.objectmentor.com/home


Agile - Review• Positives

• Software delivered• No death marches

• Dangers• Process itself becomes the goal• Placebo effect• Key point is missed

• SCRUM• Immediately useful• Becoming broadly popular

» Classes around the world» Hundreds certified


Reporting Population

• Super User » Technical skills » Understands their piece of the data» May be tactical or strategic

• Strategic User » Executive or analyst» Needs highly summarized and graphed snapshots» Limited technical skills

• Tactical User» Needs up-to-minute detailed data to do their job» Limited technical skills

Super-Users(ad-hoc)

Strategic Users( historical cubes )

Tactical Users(core & options )


Normal Report Development Process

• Demand that customer predict the future• Forecast list of needed reports• Requires visualization / no data available• Data may support 20,000,000 variations

• Design signoff in blood• Early code freeze

• Rigid Change Control• More signoffs

• Complain about scope creep• Accuse users of changing their mind

• List all changes as bugs• Falsely implies a defective product Requirements List


Agile Reporting• Bounded Chaos

• Random actions inside of the boundary

• Self Building• Developer builds 90%• User checks options• Program finishes writing itself• Program creates report

• Broad Scope• Can replace millions of fixed reports• Requirements can be hazy up front


Agile Reporting - Definitions• Report (definition)

• Any output where user can not change data in database• More than printer

• Core & Options• Core – on every report, fixed • Options – added to report based on user’s selections:

columns & totals

• Filters• Limits input data

• Medium• Web page, paper, excel, pivot table, PDF file, lists,

graphs, etc

• Measures• Changes the core - unusual


IBI Sample - ggorder2.fex

• As Installed• Units by Product• For one date

• Agile Modification• Add &sort1• Standard front-end• Add dollar value

» Immaterial to agile


Core & Option - Web Page

• Shows total sales of each product by state & sales rep• Checking no boxes: one number only


Agile ggorder2.fex

• Core only

• Boxes checked• None


Agile ggorder2.fex

• Same fex• No changes

• Boxes checked• Vendor Name• Product


Agile ggorder2.fex

• Same fex• No changes

• Boxes checked• Product

– Totals

• Store Name


Agile ggorder2.fex• Same fex

• No changes

• Boxes checked– State, City,

Store Name (totals), Vendor Name, Product

• Options– 362,889 formats

– ONE program

– Risks managed


Agile Reporting – Core & Options

• Core– Units Ordered

• Options– State– Contact– Product– Package

– Totals on: Contact, State


Core & Option Executive Demo• IBI Sales Prototype

• Used Production databases• Not standard IBI demo

» Specs and Core & Option HTML were supplied

• 4 Days development• Clear to all

• Reports• Simple list – Volumes & Dollars (agile core & option)• Delta – Change since yesterday (agile core & option)• Complex Delta – Changes of averages• 3-D Graphs• Pies with drill downs


Exclusions

• Power not demonstrated• Combined databases• Reporting Cubes• Scheduling & Mailing• Super-user interface


Risk Mitigation

• End-User reporting

– Normal recommendation

– Risks

– Mitigation for average user

– Super-users


Normal Reporting Recommendation

• Reporting is simple, use…– MS Access– Excel– Pivot Tables– WebFocus– Crystal Reports– Etc

• Training– Help– Manuals– Limited classes


Risks

• Complexity– Database structure– Queries

• All tests against production databases– Overload servers & network

• Over-reporting– Double counting data

• Under-reporting– Not including data

Deal Price

Deal Price


Mitigation• Target the skill sets

– Developers• Handle complexity• Insure accuracy• Insure efficiency• Identify possible options• Set up “Core & Option” reports

– End Users• Decide on core & options during development• Test during development• Understand the results


Products & Languages

• Effort• Easy• Varying levels of difficulty

– Brute force –vs- fits vendor’s product well

• Power• Is agile• Appears to be agile• Partly agile• Beware of pretty –vs- power


Regular File Structure - WebFocus

Pre-load ( fex )

GetCore & Options

( HTML )

Report ( fex )

Report Caster ( fex )

VariablesLists

OptionsFilters

Output Medium

Pre-setOptions, FiltersOutput Medium

Menu Item ( fex )

Pre-setOptions, FiltersOutput Medium

HTMLPDFExcel

Files

Standard Reporting ProgramRelationships


Core & Option Structure - WebFocus

• Build JavaScrips• Options & Template• Options – SQL database• Template & Fields – text file

• Hidden Fields fairly static


Report ( fex )

DEFINE FILE TRADES

Cost/P12.2CB = VOLUME * Price;

Value/P12.2 = VOLUME * STRIKE_PRICE;

Net/P12.2 = Value - Cost;

END

TABLE FILE TRADES

SUM VOLUME AS 'Volume'

Cost

Value

&sort1-* ***** add other BY statements here *****

END


The End( References )

• http://www.agilealliance.org - Agile Alliance

• http://www.jimhighsmith.com - Agile overview

• http://alistair.cockburn.us/crystal - Crystal

• http://www.controlchaos.com/Scrumo.htm - Scrum

• http://www.dsdm.org - DSDM

• http://www.poppendieck.com - LEAN

• http://www.leanconstruction.org - LEAN

• http://www.santafe.edu - Complex Adaptive

• http://www.featuredrivendevelopment.com - FDD• http://www.nebulon.com/fdd - FDD• http://www.extremeprogramming.org - EX (Extreme)• http://www.objectmentor.com/home - EX (Extreme)


Agile Manifesto (1)Follow these principles for Developing Software:

• Satisfy customerswith valuable software

• The highest priority is to satisfy the customer through early and continuous delivery of valuable software.

• Welcome change requests at any time • Making changes quickly can be a competitive

advantage• Welcome changing requirements, even late in

development. Agile processes harness change for the customer's competitive advantage.



• Deliver working software frequently • Don’t let people suffer• Deliver working software frequently, from a couple

of weeks to a couple of months, with a preference to the shorter timescale.

• Daily Contact • Customers & developers• Avoid intermediaries• Business people and developers must work

together daily throughout the project.



• Use Champions• Build projects around motivated individuals. Give

them the environment and support they need, and trust them to get the job done.

• Team Communication • Avoid long or large meetings• The most efficient and effective method of

conveying information to and within a development team is face-to-face conversation.



• Working software is primary measure• Models, documentation, use cases do not count• Working software is the primary measure of

progress

• Maintain a constant pace indefinitely• No Death Marches • Agile processes promote sustainable development.

The sponsors, developers, and users should be able to maintain a constant pace indefinitely.



• Quality enhances agility• Late and post-implementation changes easier• Continuous attention to technical excellence and

good design enhances agility.

• Self-organizing teams increase quality• Volunteers produce better results• Members look out for and help each other• The best architectures, requirements, and

designs emerge from self-organizing teams.



• Simplicity• Do less – get more• The art of maximizing the amount of work not

done -- is essential.

• Teams self-correct • Not continuously – will drive people nuts• At regular intervals, the team reflects on how to

become more effective, then tunes and adjusts its behavior accordingly.


File Naming Standards - WebFocus

• Application prefix + Name + Suffix– Drilldown can be called from many reports– Name is first report written

• Program field names – UpperLower_Case

Function Program Name Suffix

Pre-load Scr_Status_h.fex hGet Core & Options Scr_Status.htm

Report Scr_Status.fex

Drill Down Scr_Status_dr1.fex dr(+)Report Caster Scr_Status_c1.fex c(+)

Menu Item Scr_Status_m1.fex m(+)

may, 2004william (bill) myers1 agile methodologies and chaos revisited william (bill) myers cinergy...

Documents