software acquisition best practices 2004 edition · software acquisition best practices: 2004...
Post on 25-Jul-2018
214 Views
Preview:
TRANSCRIPT
Software Acquisition Best Practices:2004 Edition
Richard J. AdamsSuellen EslingerKaren L. Owens
Mary A. RichSoftware Engineering Subdivision
January 27, 2004
3rd OSD Conference on the Acquisition of Software-Intensive Systems
© 2003-2004 The Aerospace Corporation.
2
Acknowledgements
This work would not have been possible without assistance from thefollowing:
• Reviewers❖ Linda A. Abelson, Software Acquisition and Process Office❖ Peter Hantos, Software Acquisition and Process Office❖ John Cantrell, Software Architecture and Engineering Department
• Sponsor and Reviewer❖ Michael Zambrana, USAF Space and Missile Systems Center, Directorate
of Systems Engineering
• Funding Source❖ Mission-Oriented Investigative Experimentation (MOIE) Research Program
(Software Acquisition Task)
3
Outline
• Background and Definitions• Scope
❖ Software Acquisition Best Practices 2003 Reviewed❖ Scope of Software Acquisition Best Practices 2004
• Software Acquisition Best Practices 2004❖ Early Acquisition Life Cycle Phases❖ Evolutionary Acquisition
• Conclusion
4
• Definition: Best Practices are practices that people withrecognized expertise in the subject area have identified throughexperience as being significant contributors to project success
• Negative experience or positive experience may identify BestPractices❖ However, one must not be trapped by logical fallacies
• Note that Best Practices (both individually and collectively)❖ Have not necessarily undergone detailed study❖ Have almost never been analytically determined to be “best”❖ Never form an exhaustive set (There is always the possibility of more)❖ Are not static (They change with new experiences and new technologies)❖ Are dependent on the context and environment
Best Practices
PracticeNot A
PracticeA
5
Software Acquisition (SA) Best Practices
• Software Acquisition (SA) Best Practices are, therefore,practices that people with recognized software acquisitionexpertise have identified through experience as beingsignificant contributors to the successful acquisition ofsoftware-intensive systems
• The SA Best Practices presented derive from the researchteam’s collective experience in the acquisition of software-intensive space systems❖ Over 60 collective years of software acquisition experience spanning
approximately 20 years duration❖ Many additional years of experience in developing software,
managing software development projects, and leading softwareprocess improvement efforts
6
Characteristics of Space Systems (SS)
• Large software-intensive systems❖ SLOC order of magnitude: 105 onboard and 106 – 107 on the ground❖ Multi-satellite constellations❖ Multiple ground elements, frequently worldwide
• Complex combinations of hardware and software• Complex external and internal interfaces• Usually unprecedented• High reliability and integrity requirements• Developed by large teams of multiple contractors
Space Systems Software Acquisition BestPractices must support these characteristics.
7
Outline
• Background and Definitions• Best Practice Scope
❖ Software Acquisition Best Practices 2003 Reviewed❖ Scope of Software Acquisition Best Practices 2004
• Software Acquisition Best Practices 2004❖ Early Acquisition Life Cycle Phases❖ Evolutionary Acquisition
• Conclusion
8
SS SA Best Practice Scope
• Single system development contract for a software-intensive system
• Pre- and post-contract award software acquisition activitiesfor the system development contract
• Full life cycle software acquisition activities spanning thecontract award boundary❖ Software Risk Management❖ Software Systems Acquisition
9
Software Acquisition Domain
Software Engineering DomainContractor
GOVERNMENT
Pre Contract Post ContractRFP
Proposal
DevelopmentContract
Establish Program BaselineObtain Contractual InsightObtain Contractual CommitmentSelect Capable Contractor TeamProvide Contract Management Tools
Perform Technical Product ReviewsPerform Software Process ReviewsManage the Development Contract
SS SA Best Practices for aSystem Development Contract
10
SS SA Best Practice Scope
• Software acquisition activities for the full DoD and NationalSecurity Space (NSS) acquisition life cycle
• Pre- and post-contract award software acquisition activitiesfor early DoD and NSS life cycle phases
• Evolutionary acquisition
DoD and NSS Acquisition Models*
JROCJROCICDICD
NSS Space Acq Policy 03-1
PHASE B (Design Phase)Risk Reduction & Design
Development
PHASE A (StudyPhase) Concept/Architecture Dev
PHASE C (Build, Test, Launch)Acquisition & Operations Support
BA
PDRPDRSDRSDR SRR SRR
Pre-Systems Acquisition Systems Acquisition Sustainment
C
PHASE BApproval
PHASE AApproval
PHASE CApproval
CDRCDR
Pre KDP-AActivities
TechnologyTechnologyDevelopmentDevelopmentApprovalApproval
SystemSystemDevelopment &Development &DemonstrationDemonstrationApprovalApproval
A B C
Productionand
Deployment
Operations andSupport
Low-RateLow-Rate Initial Prod Initial ProdApprovalApproval
DesignDesignReadinessReadiness Review Review
CDRCDR
ConceptRefinement
TechnologyDevelopment
System Development &Demonstration
DoDI 5000.2 (12 May 2003)
KeyKeyDecisionDecisionPoints:Points:
Milestones:Milestones:
Full RateProductionApproval
IOC
IOC
11* From National Security Space Acquisition Policy #03-01, 6 October 2003.
12
Outline
• Background and Definitions• Scope
❖ Software Acquisition Best Practices 2003 Reviewed❖ Scope of Software Acquisition Best Practices 2004
• Software Acquisition Best Practices 2004❖ Early Acquisition Life Cycle Phases❖ Evolutionary Acquisition
• Conclusion
13
Concept Refinement Best Practices(Pre KDP-A Activities)
Defining:• Program life cycle• Initial Government
architecture concepts• Initial Government cost
and schedule baselines• Executable program
evolutions• Global acquisition
strategy
JROCJROCICDICD
NSS Space Acq Policy 03-1
A
Pre-Systems Acquisition
PHASE AApproval
Pre KDP-AActivities
TechnologyTechnologyDevelopmentDevelopmentApprovalApproval
A
ConceptRefinement
DoDI 5000.2 (12 May 2003)
KeyKeyDecisionDecisionPoints:Points:
Milestones:Milestones:
14
Best Practices for Defining theProgram Life Cycle
Use a software-friendlyacquisition model
• Avoid contract actions in themiddle of software developmentspirals (e.g., post System PDR)
• Develop firm basis for softwarecosting before MS B/KDP-B
• SDR level of maturity beforeMS B/KDP-B
• Selection of a single contractor atappropriate point in softwaredevelopment life cycle
• With or without production phase
ProgramLife Cycle
• Evolutionary acquisition is moresuited to large, complex software-intensive systems, such as spacesystems
Tailor the acquisition model forsoftware-intensive system
Choose software-friendly points inthe life cycle for contract actions
Example DoD and NSS Acquisition ModelsTailored for Software-Intensive Systems without Production
JROCJROCICDICD
NSS Space Acq Policy 03-1 (Adapted)
PHASE A/B Concept/Architecture Dev &
System Design
PHASE B/C Design, Development (Build, Test, Launch)& Operations Support
B/CA/B
PDRPDRSDRSDR SRR SRR
Pre-Systems Acquisition Systems Acquisition Sustainment
PHASE B/CApproval
PHASE A/BApproval
Pre KDP-AActivities
TechnologyTechnologyDevelopmentDevelopmentApprovalApproval
SystemSystemDevelopment &Development &DemonstrationDemonstrationApprovalApproval
A B C
Deployment Operations andSupport
LimitedLimitedDeploymentDeploymentApprovalApproval
DesignDesignReadinessReadiness Review Review
CDRCDR
ConceptRefinement
TechnologyDevelopment & System
Design
System Development &Demonstration
DoDI 5000.2 (12 May 2003) (Adapted)
KeyKeyDecisionDecisionPoints:Points:
Milestones:Milestones:
CDRCDR
PDRPDRSDRSDR SRR SRR
FullDeploymentApproval
IOC
IOC
15
16
Best Practices for Developing the InitialGovernment Architecture Concepts
Include software in evaluation ofarchitecture concepts
• Evaluate software evolution andgrowth capability
• Include software in life cycle costanalysis (COTS software refresh,legacy and new software re-engineering and maintenance)
GovernmentArchitecture
Concepts
Perform software-inclusivearchitecture trade studies
• Able to grow with each successiveevolution with little expected rework
• Able to integrate each successiveevolution with previous evolutions(and legacy system, as applicable)
• With system architecture trades• Identify and address critical HW/SW
architecture issues• Include major legacy components
and COTS software
Select a set of integratedHW/SW architecture concepts
17
Best Practices for Developing the InitialGovernment Cost and Schedule Baseline
Determine realistic SW effort & costestimates for each evolution
• Include COTS, reuse and newlydeveloped software
• Include tasks not reflected in costmodels (e.g., integration of SWcomponents costed separately, COTS)
• Include all software effort inschedule
• Never compress softwareschedule >20% off nominal*
Gov’t. Cost And Schedule
Baseline
Determine realistic SW scheduleestimates for each evolution
Determine realistic SW sizeestimates for each evolution
• Use Gov’t. HW/SW architecture concept• Include all SW functionality and
infrastructure needed• Use historical data from similar past
programs & early concept study data
* Software Program Manager’s Network, The ProgramManager’s Guide to Software Acquisition Best Practices,Version 2.31, p. 22.
18
Best Practices for DefiningExecutable Program Evolutions
Consider SW implications whendefining evolution capabilities
• Analyze feasibility of overlappingsoftware development schedulesfor closely spaced evolutions
• Avoid plans that require developingsubsequent evolutions on unknownsoftware baselines
• Analyze feasibility of COTS refreshand legacy SW upgrade schedules
ExecutableProgram
Evolutions
• Analyze feasibility of developing therequired software for each evolution
• Based on realistic software size,effort, cost and scheduleestimates
• Include software cost andschedule estimation risk
• Analyze feasibility of integrating thesoftware in each evolution with allprevious evolutions (and legacysystem(s), as applicable)
• Based on integrated hardware/software architecture
• Analyze impacts of COTS softwarerefresh and legacy softwareupgrades
Consider SW implications whendefining evolution schedules
19
Develop plans for computersystem technology insertion
Develop plans for evaluation ofcontractor software capability
• Perform a Government evaluationof contractor team softwarecapability
• Prior to or part of selection of asingle development contractor
GlobalAcquisition
Strategy
• Include COTS HW and SW refresh ineach successive evolution
• Understand new computer HW & SWtechnologies needed for eachevolution and study their readiness
• Plan for managing multiple baselines(operations and development)
• Plan for integrating softwaremaintenance actions on operationalevolutions into evolutions underdevelopment
Develop plans for softwaresupport
Best Practices for Developing theGlobal Acquisition Strategy
20
Concept/Technology/ArchitectureDevelopment (Phase A/B) Best Practices
Principal objective of Phase A/Bcontract(s)* is to develop theinformation needed for theGovernment to:
❖ Solidify the programdefinition to establish anexecutable program
❖ Update the globalacquisition strategy,including acquisition plansand products for this andall future evolutions
NSS Space Acq Policy 03-1 (Adapted)
PHASE A/B Concept/Architecture Dev &
System Design
B/CA/B
SDRSDR SRR SRR
Pre-Systems Acquisition
PHASE B/CApproval
PHASE A/BApproval
TechnologyTechnologyDevelopmentDevelopmentApprovalApproval
SystemSystemDevelopment &Development &DemonstrationDemonstrationApprovalApproval
A B
DoDI 5000.2 (12 May 2003) (Adapted)
KeyKeyDecisionDecisionPoints:Points:
Milestones:Milestones:
SDRSDR SRR SRR
TechnologyDevelopment & System
Design
* Space systems usually have multipleparallel contracts in this phase, withselection of a single development contractorin the next phase (B/C).
21
SS SA Best Practicesfor a Phase A/B Contract
Contractor
Software Acquisition DomainGOVERNMENT
Software Engineering Domain
Pre Contract Post Contract
Establish Requirements BaselineDevelop System Architecture ConceptReduce Software Risk
Manage the Phase A/B Contract
RFP
ProposalPhase A/BContract
22
Include software in Gov’t. systemperformance requirements
Contract for delivery of SW-inclusive reqs. specifications
• Require System and SegmentSpecifications as CDRL items
• Use System/SubsystemSpecification DID (DI-IPSC-81431a)
RequirementsBaseline
Best Practices for Establishing theRequirements Baseline
• Specialty engineering, especiallyRMA
• Key Performance Parameters• Open system architecture• Design for evolution and growth
23
Contract for softwarearchitecture trade studies
Contract for delivery of systemarchitecture
• Require system architecture as aCDRL item
• Require an integrated HW/SWarchitecture, defined by multiplearchitecture views
• Include newly developed, reuse andCOTS software
SystemArchitectural
Design
• With system architecture trades• Include major software legacy
components and COTS software
Best Practices for Developing theSystem Architectural Design
24
Contract for software productrisk reduction
Contract for software processrisk reduction
• Require delivery of SoftwareDevelopment Plan (DID DI-IPSC-81427a)
• Require compliance with robustsoftware development standard
• Enable contractor team to preparefor software capability evaluation
SW DevelopmentRisk Reduction
• Studies/prototyping of high riskareas for software, e.g.
•Mission processing algorithms•Mission planning concepts
• Simulation development• Increase readiness level of computer
HW and SW technologies
Best Practices for ReducingSoftware Development Risk
25
Best Practices for Managingthe Phase A/B Contract
• Government software acquisitionpersonnel with technical expertisein software product and processengineering must participate
Participate with contractor insoftware risk reduction
Ensure contractor(s) defineintegrated HW/SW architecture
• Software systems engineers(contractor and Government)must participate with contractorand Gov’t. systems engineers
• Software systems engineers(contractor and Government)must participate with contractorand Gov’t. systems engineers
Ensure contractor(s) definesoftware-inclusive reqs. specs.
Managing the Contract
26
Evolutionary Acquisition Strategy - 1
Global Acquisition
Strategy
A/B B/C (O&S)
B/CIOC
Increment 3
PreA
A/B B/C (O&S)
B/CA/B
IOC
Increment 1
A/B B/C (O&S)
B/CIOC
Increment 2
Ongoing or near term
Future planning
Feedback
Acquisition Planning
27
Evolutionary Acquisition Strategy - 2
PreA
A/B B/C (O&S)
B/CA/B
IOC
Future planning
A/B B/C (O&S)
B/CIOC
Increment 3
Global Acquisition
Strategy
Increment 1
A/B B/C (O&S)
B/CIOC
Increment 2
Ongoing or near term
Feedback
Feedback
Acquisition Planning
28
Update SW-inclusive programbaseline
Update definition of SW-friendlyevolutions
• Evolution capabilities, schedulesand integration strategies
• COTS software refresh and legacysoftware upgrades
• Software support strategy• Contractor team software capability
evaluations• Software technology insertion• Software transition to operations
Updated Global Acquisition
Strategy
• Software-inclusive systemrequirements
• Integrated HW/SW architecture• Realistic software size, effort, cost &
schedule estimates for eachevolution
Update software-specific plans
Best Practices for Updating theGlobal Acquisition Strategy
29
Software Acquisition RiskManagement
• Integrate software acquisition withthe system acquisition process
• From capability needsidentification through systemretirement
• Especially during earlyacquisition life cycle phases
• Continuous software acquisitionrisk management
• Across the entire acquisitionlife cycle
• Across all evolutions• Within each ongoing evolution
• Program level risk management andcontractor development riskmanagement are necessary but notsufficient
Software Systems Acquisition
Best Practices that Span theDoD and NSS Acquisition Life Cycle
Software Acquisition Risk ManagementSoftware Systems Acquisition
PHASE A/B PHASE B/CPre KDP-A
30
Outline
• Background and Definitions• Scope
❖ Software Acquisition Best Practices 2003 Reviewed❖ Scope of Software Acquisition Best Practices 2004
• Software Acquisition Best Practices 2004❖ Early Acquisition Life Cycle Phases❖ Evolutionary Acquisition
• Conclusion
31
Conclusion
• Software acquisition best practices do not guarantee success❖ They are not a panacea!
• Using best practices, however, can reduce risk in complexsoftware-intensive system acquisitions
• Evolutionary acquisition, in particular, is a complex strategythat requires careful planning and execution in order toachieve its anticipated benefits
• Software acquisition best practices will be most effectivelyimplemented if done in the context of a software acquisitionprocess improvement program❖ Based on experiences with software development
Section 804 of the FY03 Defense Authorization Actrequires the establishment of software acquisition
process improvement programs.
32
Back-Up Charts
• Software Acquisition Best Practices 2003• Acronym List• Author Contact Information
33
Perform software architecture-inclusive trade studies
Determine realistic, independentbaseline software estimates
• Size, effort, cost and schedule• COTS, reuse and newly developed• Tasks not reflected in cost models• Realism especially critical for
evolutionary acquisition
• Specialty engineering, esp. RMA• Key Performance Parameters• Open system architecture
ProgramBaseline
• With system architecture trades• Include major legacy components• Supports Government software
architecture baseline selection
Include software in systemperformance requirements
Best Practices forEstablishing the Program Baseline
34
Best Practices forObtaining Contractual Insight
• In addition to system reviews
ContractualInsight
• Highest risk reduction potential:• Plans (development, build,
transition)• Requirements & Architecture• Test plans, procedures & reports• Metrics reports• Delivery, installation & maintenance
documentation• Use electronic delivery
Require key software technical& management deliverables
Require software level technical& management reviews
Require timely electronicaccess to all software products
• Requirements• Architecture, Design• Implementation (including code)• Integration and Verification Testing• Intermediate and Final Products
35
Mandate compliance with robustfull life cycle SW dev. standard
• Include commitment in IntegratedMaster Plan (IMP)
ContractualCommitment
• For example, EIA/IEEE J-STD-016
Require contractor commitmentto Software Development Plan
Best Practices forObtaining Contractual Commitment
36
Evaluate software capability aspart of source selection
Capable Software Contractor Team
• Evaluate software capability ofofferor teams
• Individual team memberevaluation insufficient
• Evaluate software capability/processes as subfactor
• Under Mission Capability factor• Weight according to software
risk• Evaluate teams’ proposed software
processes• Corporate and past project
process evaluation insufficient
Best Practices for Selecting aCapable Software Contractor Team
• Suspect extremes of productivity,COTS & reuse, & low lines of code
Evaluate realism of cost andschedule bids
Evaluate software architecturewith system design
• Evaluate major HW/SW architectureissues (e.g., space-ground trades,reuse of legacy components)
37
Incentivize software quality,*not just cost and schedule
• Relate results and improvementactions directly to award fee
Tools forContract
Management
• Use award and incentive fee plans• Reward adherence to
• Defined software processes• Software process improvement
• Reward timely and adequateresponse to Government comments
• Reward low rework rates• Reward meeting RMA requirements
post delivery/launch
Mandate periodic team softwarecapability appraisals
Best Practices for Providing Toolsfor Contract Management
* Quality in this context is producing workproducts that do not require rework insuccessor activities.
38
Perform in-depth technicalreviews of software products
• Begin at the build level• Focus on areas of highest risk• Focus on early performance
analysis results and meeting KPPs
Technical Product Reviews
• IPTs, TIMs, working groups, peerreviews, etc.
• Software Level Technical Reviews• High risk/critical software products• Key software technical deliverables• Focus on areas of highest risk
Monitor software integrationand verification adequacy
Best Practices for PerformingTechnical Product Reviews
Include users/operators in alltechnical review activities
• Focus on operational suitability ofevolving software-intensive system
39
Best Practices for PerformingSoftware Process Reviews
SoftwareProcessReviews
• Review team’s adherence to definedsoftware processes
• Identify adherence deficiencies• Assist in deficiency correction
• Evaluate effectiveness of definedSW processes
• Identify process deficiencies• Assist with process
improvement• Level 2 & 3 CMMI®/CMM® adherence
for an individual team member maynot be sufficient*
Review effectiveness ofcontractor team’s SW processes
Perform periodic team softwarecapability appraisals
• During contract performance• Support for significant program or
award fee milestones
* CMM and CMMI are registered trademarksof Carnegie Mellon University.
40
Use incentive/award feesaggressively
• Especially RMA
Managing the Contract
• Motivate good software practices• Focus on quality
Ensure satisfaction of software–inclusive requirements
Best Practices forManaging the Development Contract
Apply proactive quantitativemanagement
• Ensure a comprehensive software/system metrics program balancedacross information categories
• Include leading quality indicators(e.g., rework)
• Perform cross-metric analysis• Earned value alone is insufficient
Perform periodic independentassessments
• Support for significant program oraward fee milestones
• Act aggressively on findings
41
Acronyms and Abbreviations - 1
Acq AcquisitionCDR Critical Design ReviewCDRL Contract Data Requirements ListCMM® Capability Maturity Model®
CMMI® Capability Maturity Model® IntegrationSM
COTS Commercial Off the ShelfDB DatabaseDev DevelopmentDID Data Item DescriptionDoD Department of DefenseDoDI DoD InstructionEIA Electronic Industries AllianceFY Fiscal YearGov’t. GovernmentGUI Graphical User InterfaceHW HardwareIEEE Institute of Electrical and Electronics Engineers
42
Acronyms and Abbreviations - 2
IMP Integrated Management PlanIPT Integrated Product TeamIOC Interim Operational CapabilityJ JointKDP Key Decision PointKPP Key Performance ParameterMOIE Mission-Oriented Investigation and ExperimentationMS MilestoneNSS National Security SpaceO&S Operations and SupportOSD Office of the Secretary of DefensePDR Preliminary Design ReviewRFP Request for ProposalRMA Reliability, Maintainability, AvailabilitySA Software AcquisitionSDP Software Development PlanSDR System Design Review
43
Acronyms and Abbreviations - 3
SLOC Source Lines of CodeSM Service MarkSRR System Requirements ReviewSS Space SystemSTD StandardSW SoftwareTIM Technical Interchange MeetingUSAF United States Air Force
44
Author Contact Information
• Richard J. Adams❖ Senior Engineering Specialist❖ Software Engineering
Subdivision, The AerospaceCorporation
❖ (310) 336-2907❖ Richard.J.Adams@aero.org
• Suellen Eslinger❖ Distinguished Engineer❖ Software Engineering
Subdivision, The AerospaceCorporation
❖ (310) 336-2906❖ Suellen.Eslinger@aero.org
• Karen L. Owens❖ Senior Engineering Specialist❖ Software Engineering
Subdivision, The AerospaceCorporation
❖ (310) 336-5909❖ Karen.L.Owens@aero.org
• Mary A. Rich❖ Principal Director❖ Software Engineering
Subdivision, The AerospaceCorporation
❖ (310) 336-5313❖ Mary.A.Rich@aero.org
top related