1
Value-Based Software Engineering I: Motivation and Key Practices
LiGuo HuangComputer Science and Engineering
Southern Methodist University
2
Outline• Motivation and definitions• Understanding sources of value• Seven key practices
1. Benefits Realization Analysis2. Stakeholders’ Value Proposition Elicitation and
Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity Management5. Concurrent System and Software Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
• Conclusions; references
3
Software Testing Business Case
• Vendor proposition– Our test data generator will cut your test costs in half– We’ll provide it to you for 30% of your test costs– After you run all your tests for 50% of your original cost,
you are 20% ahead• Any concerns with vendor proposition?
4
Software Testing Business Case• Vendor proposition
– Our test data generator will cut your test costs in half– We’ll provide it to you for 30% of your test costs– After you run all your tests for 50% of your original cost,
you are 20% ahead• Any concerns with vendor proposition?
– Test data generator is value-neutral*– Every test case, defect is equally important– Usually, 20% of test cases cover 80% of business case
* As are most current software engineering techniques
VERs for Value-Based Testing vs. Value-Neutral Testing (Bullock, 2000)
% of Valuefor
CorrectCustomer
Billing
Customer Type
100
80
60
40
20
5 10 15
Automated test generation tool - all tests have equal value
Bullock data– Pareto distribution
ROI: Value-Neutral ATG vs. Pareto Analysis (1)
-1.5
-1
-0.5
0
0.5
1
1.5
2
0 10 20 30 40 50 60 70 80 90 100
% Tests Run
Ret
urn
On
Inve
stm
ent (
RO
I)
ATG Testing Pareto Testing
ROI: Value-Neutral ATG vs. Pareto Analysis (2)
ATG Testing Pareto Testing % of Tests Run
Cost ($K)
Value ($K)
Net Value ($K) ROI Cost
($K) Value ($K)
Net Value ($K) ROI
0 1300 0 -1300 -1.0 1000 0 -1000 -1.0 10 1350 400 -950 -.70 1100 2560 1460 +1.33 20 1400 800 -600 -.43 1200 3200 2000 1.67 40 1500 1600 100 +.07 1400 3840 2440 1.74 60 1600 2400 800 .50 1600 3968 2368 1.48 80 1700 3200 1500 .88 1800 3994 2194 1.21 100 1800 4000 2200 1.22 2000 4000 2000 1.0
Assumptions: • $1M development investments in customer billing system by beginning of testing• ATG tool costs $300K and will reduce the costs by 50% as promised• The business case for the system will produce $4M in business value in return for
$2M investment• The business case will produce a similar 80:20 distribution for the remaining 14
customer types
8
Motivation for Value-Based SE• Current SE methods are basically value-neutral
– Every requirement, use case, object, test case, and defect is equally important
– Object oriented development is a logic exercise– “Earned Value” Systems don’t track business value– Separation of concerns: SE’s job is to turn requirements into
verified code– Ethical concerns separated from daily practices
• Value – neutral SE methods are increasingly risky– Software decisions increasingly drive system value– Corporate adaptability to change achieved via software
decisions– System value-domain problems are the chief sources of
software project failures
9
The “Separation of Concerns” Legacy
• “The notion of ‘user’ cannot be precisely defined, and therefore has no place in CS or SE.”
- Edsger Dijkstra, ICSE 4, 1979
• “Analysis and allocation of the system requirements is not the responsibility of the SE group but is a prerequisite for their work”
- Mark Paulk at al., SEI Software CMM* v.1.1, 1993
*Capability Maturity Model
10
Resulting Project Social Structure
SOFTWARE
MGMT.
AERO. ELEC. G & C
MFG.
COMM PAYLOAD
I wonder whenthey'll give us ourrequirements?
11
Why Software Projects Fail
12
20% of Fires Cause 80% of Property Loss:Focus Fire Dispatching on These?
100
80
60
40
20
% of Property
Loss
% of Fires20 40 60 80 100
13
Need to Consider Fairness and Ethics:-Rawls’ Theory of Justice (1971)
• Fair rules of conduct• Principles of justice• Participants and obligations
– Provider (developer)– Buyer (acquire)– User(s)– Penumbra (general public)
• Negotiate mutually satisfactory (win-win) agreements
14
Rawls’ Theory of Justice - II• Fair rules of conduct
– Negotiation among interested parties– Veil of ignorance (about what affects whom)– Rationality
• Principles– Least Advantaged - don’t increase harm to them
• Harm = probability x magnitude (~risk exposure)– Risking Harm - don’t risk increasing harm
• Don’t use “low-threat” software in “high-threat” context
– Publicity test - defensible with honor before an informed public
• Use for difficult cost-benefit tradeoffs
15
Penumbra Negotiation Example: Fire Dispatching System
• Dispatch to minimize value of property loss– Neglect safety, least-advantaged property owners
• English-only dispatcher service– Neglect least-advantaged immigrants
• Minimal recordkeeping– Reduced accountability
• Tight budget; design for nominal case– Neglect reliability, safety, crisis performance
16
Key Definitions
• Value (from Latin “valere” – to be worth1. A fair or equivalent in goods, services, or money2. The monetary worth of something3. Relative worth, utility or importance
• Software validation (also from Latin “valere”)– Validation: Are we building the right product?– Verification: Are we building the product right?
17
Conclusions So Far• Value considerations are software success-
critical• “Success” is a function of key stakeholder
values– Risky to exclude key stakeholders
• Values vary by stakeholder role• Non-monetary values are important
– Fairness, customer satisfaction, trust• Value-based approach integrates ethics
into daily software engineering practice
18
Outline• Motivation and definitions• Understanding sources of value• Seven key practices
1. Benefits Realization Analysis2. Stakeholders’ Value Proposition Elicitation and
Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity Management5. Concurrent System and Software Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
• Conclusions; references
19
What is “Value”
“relative worth, utility, or importance”− broader dictionary definition
20
The Model-Clash Spider Web: Master Net- Stakeholder value propositions (win conditions)
21
Maslow Human Need HierarchyA. Maslow, Motivation and Personality, 1954.
Self-Actualization
Esteem and Autonomy
Belongingness and love
Safety and Security
Physiological (Food and Drink)
22
Maslow Need Hierarchy• Satisfied needs aren’t motivators• Unsatisfied lower-level needs dominate higher-level needs• Management implications
– Create environment and subculture which satisfies lower-level needs
• Stability• Shared values, community• Match to special needs
– Tailor project objectives, structure to participants’ self-actualization priorities
23
People Self-Actualize in Different Ways
• Becoming a Better Manager• Becoming a Better Technologist• Helping Other Developers• Helping Users• Making People Happy• Making People Unhappy• Doing New Things• Increasing Professional Stature
24
Projecting Yourself Into Others’ Win Situations
• Do unto others
.. As you would have others do unto you
Counterexample: The Golden Rule
25
Projecting Yourself Into Others’ Win Situations
• Build computer systems to serve users and operators
.. Assuming users and operators like to write programs, and know computer science
•Computer sciences world (compilers, OS, etc.)–Users are programmers
•Applications world–Users are pilots, doctors, tellers
• Do unto others
.. As you would have others do unto you
Counterexample: The Golden Rule
26
Value-Based Software Engineering
Definition: “the explicit concern with value
concerns in the application of science and mathematics by which the properties of computer software are made useful to people”
27
VBSE Agenda• Value-based requirements engineering• Value-based architecting• Value-based design and development• Value-based verification and validation• Value-based planning and control• Value-based risk management• Value-based quality management• Value-based people management• A theory of value-based software engineering
28
7 Key Elements of VBSE1. Benefits Realization Analysis2. Stakeholders’ Value Proposition
Elicitation and Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity
Management5. Concurrent System and Software
Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
29
DMR/BRA* Results Chain
INITIATIVE OUTCOMEOUTCOME
Implement a new orderentry system
ASSUMPTION
Contribution Contribution
Order to delivery time isan important buying criterion
Reduce time to process order
Reduced order processing cycle(intermediate outcome)
Increased sales
Reduce time to deliver product*DMR Consulting Group’s Benefits Realization Approach
30
Usage of Results Chain
• Identify non-software initiatives related to value-producing results
• Identify additional success-critical stakeholders (SCS)
31
7 Key Elements of VBSE1. Benefits Realization Analysis2. Stakeholders’ Value Proposition
Elicitation and Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity
Management5. Concurrent System and Software
Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
32
Stakeholder Value Proposition Elicitation
• Start with Results Chain• Interview Success-Critical Stakeholders• Model Clash Spider Web as a top-level checklist• Synthetic experience techniques
– Prototypes, scenarios, stories• Shared Vision Document
– Elevator description– Results chain– Success-critical stakeholders and roles– System Block Diagram
33
Stakeholder Value Proposition Reconciliation
• Expectation Management– Be aware of the number of potential value proposition conflicts– Lessons learned retrospectives– Well-calibrated cost model– “simplifier and complicator”
• Visualization and trade-off analysis– Prototypes; scenarios, estimation models
• Prioritization– Pairwise comparison, scale-of-ten ratings of importance and difficulty
• Groupware– Easywinwin
• Business case analysis– ROI
34
EasyWinWin OnLine Negotiation Steps
35
Red cells indicate lack of Red cells indicate lack of consensus. consensus. Oral discussion of cell Oral discussion of cell graph reveals unshared graph reveals unshared information, unnoticed information, unnoticed assumptions, hidden assumptions, hidden issues, constraints, etc.issues, constraints, etc.
Tradeoffs among Cost, Schedule, and Reliability – 100K-SLOC Project
0123456789
0 10 20 30 40 50
Development Time (Months)
Cos
t ($M
)
(VL, 1)
(L, 10)
(N, 300)
(H, 10K)
(VH, 300K)
(RELY, MTBF (hours))
-- Cost/Schedule/RELY: pick any two” points
•For 100-KSLOC set of features•Can “pick all three” with 77-KSLOC set of features
37
7 Key Elements of VBSE1. Benefits Realization Analysis2. Stakeholders’ Value Proposition
Elicitation and Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity
Management5. Concurrent System and Software
Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
38
Example of Business Case AnalysisROI= (Benefits-Costs)/Costs
Time
Return on Investment
-1
1
2
3 Option B
Option A
39
Example of Business Case AnalysisROI= (Benefits-Costs)/Costs
Time
Return on Investment
-1
1
2
3 Option B-Rapid
Option B
Option A
40
Unquantifiable Factors
• Benefits– Multiple criterion decision making– Stakeholder utility functions
• Uncertainties and risk– Uncertain assumptions– Buy information to reduce risk
41
7 Key Elements of VBSE1. Benefits Realization Analysis2. Stakeholders’ Value Proposition
Elicitation and Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity
Management5. Concurrent System and Software
Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
42
Understanding People’s Utility Functions
• Risk-averse people• Reconcile people’s utility functions
43
How Can Risk Management Help You Deal With Risks?
• Buying information• Risk avoidance• Risk transfer• Risk reduction• Risk acceptance
44
What Else Can Risk Management Help You Do?• Determine “How much is enough?” for your products and
processes– Functionality, documentation, prototyping, COTS
evaluation, architecting, testing, formal methods, agility, discipline, …
– What’s the risk exposure of doing too much?– What’s the risk exposure of doing too little?
• Tailor and adapt your life cycle processes– Determine what to do next (specify, prototype, COTS
evaluation, business case analysis)– Determine how much of it is enough– Examples: Risk-driven spiral model and extensions
(win-win, anchor points, RUP, MBASE, CeBASE Method)
• Get help from higher management– Organize management reviews around top-10 risks
45
Example Large-System Risk Analysis:How Much Architecting is Enough?
• Large system involves subcontracting to over a dozen software/hardware specialty suppliers
• Early procurement package means early start– But later delays due to inadequate architecture– And resulting integration rework delays
• Developing thorough architecture specs reduces rework delays– But increases subcontractor startup delays
46
How Soon to Define Subcontractor Interfaces?Risk exposure RE = Prob(Loss) * Size(Loss)
-Loss due to rework delays
Time spent defining & validating architecture
RE =P(L) * S(L)
Many interface defects: high P(L)Critical IF defects: high S(L)
Few IF defects: low P(L)Minor IF defects: low S(L)
47
How Soon to Define Subcontractor Interfaces?- Loss due to rework delays
- Loss due to late subcontact startups
Time spent defining & validating architecture
RE =P(L) * S(L)
Few delays: low P(L)Short Delays: low S(L)
Many delays: high P(L)Long delays: high S(L)
Many interface defects: high P(L)Critical IF defects: high S(L)
Few IF defects: low P(L)Minor IF defects: low S(L)
48
Time spent defining & validating architecture
RE =P(L) * S(L)
Many delays: high P(L)Long delays: high S(L)
SweetSpot
Many interface defects: high P(L)Critical IF defects: high S(L)
Few IF defects: low P(L)Minor IF defects: low S(L)
How Soon to Define Subcontractor Interfaces?- Sum of Risk Exposures
Few delays: low P(L)Short delays: low S(L)
49
How Soon to Define Subcontractor Interfaces?- Very Many Subcontractors
RE =P(L) * S(L)
Higher P(L), S(L): many more IF’s
Mainstream Sweet
Spot
Many-SubsSweetSpot
Time spent defining & validating architecture
50
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60
Percent of Time Added for Architecture and Risk Resolution
Perc
ent o
f Tim
e A
dded
to O
vera
ll Sc
hedu
le
How Much Architecting Is Enough?-A COCOMO II Analysis
Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution
Added Schedule Devoted to Rework(COCOMO II RESL factor)
Total % Added Schedule
10000KSLOC
100 KSLOC
10 KSLOC
Sweet Spot
Sweet Spot Drivers:
Rapid Change: leftward
High Assurance: rightward
51
7 Key Elements of VBSE1. Benefits Realization Analysis2. Stakeholders’ Value Proposition
Elicitation and Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity
Management5. Concurrent System and Software
Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
52
Sequential Engineering Neglects Risk
$100M
$50M
Arch. A:Custommany cache processors
Arch. B:ModifiedClient-Server
1 2 3 4 5
Response Time (sec)
Original Spec After Prototyping
53
Concurrent Process Models
• Spiral process models vs. waterfall process models
• Milestone pass/fail criteria• Process maturity model
– CMM → CMMI
54
7 Key Elements of VBSE1. Benefits Realization Analysis2. Stakeholders’ Value Proposition
Elicitation and Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity
Management5. Concurrent System and Software
Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
55
7 Key Elements of VBSE1. Benefits Realization Analysis2. Stakeholders’ Value Proposition
Elicitation and Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity
Management5. Concurrent System and Software
Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
56
Sources of Changes
• Changes in technology• Changes in marketplace opening up new
opportunities for value creation
57
Change As Opportunity: Agile Methods
• Architecture-based– Identify product’s most likely sources of changes– Evolutionary requirements– Information hiding modularity– Schedule as Independent Variable (SAIV)
• Refactoring-based– Simple design and refactoring vs. big design up front
• Continuous customer interaction• Short value - adding increments• Tacit interpersonal knowledge
– Stories, planning game, pair programming– Explicit documented knowledge expensive to change
58
Five Critical Decision Factors• Represent five dimensions• Size, Criticality, Dynamism, Personnel, Culture
Personnel
Dynamism (% Requirements-change/month)
Culture (% thriving on chaos vs. order)
Size (# of personnel)
Criticality (Loss due to impact of defects)
5030
105
1
90
70
50
30
10
3
10
30
100
300
35
30
25
20
15
Essential Funds Discretionary
Funds Comfort
Single Life
Many Lives
(% Level 1B) (% Level 2&3)
0
10
20
30
40
Agile
Plan-driven
59
Economic Value of Adaption to Change
• Change-anticipatory modular design– Real options– Buy information to reduce wrong set of changes
60
Outline• Motivation and definitions• Understanding sources of value• Seven key practices
1. Benefits Realization Analysis2. Stakeholders’ Value Proposition Elicitation and
Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity Management5. Concurrent System and Software Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
• Conclusions; references
61
Conclusions• Marketplace trends favor transition to VBSE paradigm
– Software a/the major source of product value– Software the primary enabler of adaptability
• VBSE involves 7 key elements1. Benefits Realization Analysis2. Stakeholders’ Value Proposition Elicitation and
Reconciliation3. Business Case Analysis4. Continuous Risk and Opportunity Management5. Concurrent System and Software Engineering6. Value-Based Monitoring and Control7. Change as Opportunity
• Processes for implementing VBSE emerging– MBASE, CMMI, DMR/BRA, Balanced Scorecard, RUP
extensions, Strategic Design, Agile Methods
62
ReferencesC. Baldwin & K. Clark, Design Rules: The Power of Modularity, MIT Press, 1999.B. Boehm, “Value-Based Software Engineering,” ACM Software Engineering Notes, March 2003.B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E. Horowitz, R. Madachy, D. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, 2000.B. Boehm and L. Huang, “Value-Based Software Engineering: A Case Study, Computer, March 2003, pp. 33-41.
B. Boehm & K. Sullivan, “Software Economics: A Roadmap,” The Future of Software Economics, A. Finkelstein (ed.), ACM Press, 2000.B. Boehm and R. Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Addison Wesley, 2003 (to appear).
S. Faulk, D. Harmon, and D. Raffo, “Value-Based Software Engineering (VBSE): A Value-Driven Approach to Product-Line Engineering,” Proceedings, First Intl. Conf. On SW Product Line Engineering, August 2000.
R. Kaplan & D. Norton, The Balanced Scorecard: Translating Strategy into Action, Harvard Business School Press, 1996.
63
D. Reifer, Making the Software Business Case, Addison Wesley, 2002.
K. Sullivan, Y. Cai, B. Hallen, and W. Griswold, “The Structure and Value of Modularity in Software Design,” Proceedings, ESEC/FSE, 2001, ACM Press, pp. 99-108.
J. Thorp and DMR, The Information Paradox, McGraw Hill, 1998.
Economics-Driven Software Engineering Research (EDSER) web site: www.edser.org
MBASE web site : sunset.usc.edu/research/MBASE