systems engineering technical review (setr): navy(setr ... navy’s...systems engineering technical...
TRANSCRIPT
Systems Engineering Technical Review Systems Engineering Technical Review (SETR): Navy’s Acquisition and(SETR): Navy’s Acquisition and(SETR): Navy s Acquisition and (SETR): Navy s Acquisition and
Development Process Development Process andandandand
Software Lessons LearnedSoftware Lessons Learned
Naval Air Systems CommandSystems Engineering Department
Software Engineering & Acquisition Management Division
11
Distribution Statement A. Approved for public release; distribution is unlimited.
Processes, Models, and Processes, Models, and , ,, ,FrameworksFrameworks
The following key processes, models, and The following key processes, models, and frameworks are synergistic and when used frameworks are synergistic and when used together provide for efficiency and effectivenesstogether provide for efficiency and effectivenesstogether provide for efficiency and effectiveness together provide for efficiency and effectiveness of systems engineering and software acquisition of systems engineering and software acquisition and development:and development:
Systems Engineering Technical Review (SETR)Systems Engineering Technical Review (SETR)Systems Engineering Technical Review (SETR) Systems Engineering Technical Review (SETR) Acquisition frameworkAcquisition frameworkBasic Systems EngineeringBasic Systems EngineeringIEEE/EIA 12207IEEE/EIA 12207 Software life cycle processesSoftware life cycle processesIEEE/EIA 12207 IEEE/EIA 12207 Software life cycle processesSoftware life cycle processesCapability Maturity Model Integration (CMMICapability Maturity Model Integration (CMMI®®) Best ) Best Practices frameworkPractices framework
22
Processes, Models, and Processes, Models, and , ,, ,Frameworks cont.Frameworks cont.
While each of the four is normally taught While each of the four is normally taught separately, the establishment of a software separately, the establishment of a software engineering environment consistent with all fourengineering environment consistent with all fourengineering environment consistent with all four engineering environment consistent with all four provides for an infrastructure that can be provides for an infrastructure that can be measured for effectiveness and improved to measured for effectiveness and improved to ppprovide the warfighter with better quality provide the warfighter with better quality products faster and cheaper products faster and cheaper –– BEST VALUEBEST VALUETh b d b t ti th h tTh b d b t ti th h tThey are based on best practices throughout They are based on best practices throughout industry, government, and academia and are industry, government, and academia and are designed to work with each otherdesigned to work with each other
33
designed to work with each other designed to work with each other
i S i ii S i iBasic Systems EngineeringBasic Systems Engineering
44
Systems EngineeringSystems Engineeringy g gy g g
A system is an integrated composite of people, A system is an integrated composite of people, d t d th t id bilitd t d th t id bilitproducts, and processes that provide a capability products, and processes that provide a capability
to satisfy a stated need or objectiveto satisfy a stated need or objectiveSystems Engineering (SE) is the effectiveSystems Engineering (SE) is the effectiveSystems Engineering (SE) is the effective Systems Engineering (SE) is the effective application of scientific and engineering efforts application of scientific and engineering efforts to transform an operational need into a defined to transform an operational need into a defined system configuration through the topsystem configuration through the top--down down iterative process of requirements definition, iterative process of requirements definition, functional analysis and allocation synthesisfunctional analysis and allocation synthesisfunctional analysis and allocation, synthesis, functional analysis and allocation, synthesis, optimization, design, test, and evaluationoptimization, design, test, and evaluation
55
Source: NAVAIR ACQUISITION GUIDE 2006/2007, 20 Sep 2006.
S stems Enginee ing P ocessS stems Enginee ing P ocessSystems Engineering ProcessSystems Engineering ProcessDefine customer needs and required Define customer needs and required f ti litf ti litfunctionality functionality Plan the project Plan the project Document the requirementsDocument the requirements
PlanningPlanning
Requirements Requirements qqValidate and baseline requirementsValidate and baseline requirements
Develop the design based on the Develop the design based on the requirementsrequirements
qqPhasePhase
Design PhaseDesign PhaseqqValidate that design meets all requirements and is Validate that design meets all requirements and is baselined baselined
Produce the item based on the designProduce the item based on the design
Design PhaseDesign Phase
Coding/Coding/ImplementationImplementation
Perform system validationPerform system validationEnsure all requirements met and the required Ensure all requirements met and the required functionality performs as expectedfunctionality performs as expected
TestingTesting
66
Based International Council on Systems Engineering (INCOSE) Based International Council on Systems Engineering (INCOSE) definitiondefinition
Systems Engineering TechnicalSystems Engineering TechnicalSystems Engineering Technical Systems Engineering Technical Review ProcessReview Process
77
SETRSETRWhat is the SETR?What is the SETR?
Systems Engineering Technical Review was developed by NAVAIR and has been Systems Engineering Technical Review was developed by NAVAIR and has been adapted for use within the entire Navyadapted for use within the entire NavyAn iterative program timeline that maps the technical reviews to the acquisitionAn iterative program timeline that maps the technical reviews to the acquisitionAn iterative program timeline that maps the technical reviews to the acquisition An iterative program timeline that maps the technical reviews to the acquisition process described in DOD 5000 documentationprocess described in DOD 5000 documentation
Aligned with DODI 5000.02Aligned with DODI 5000.02Policy and process described in NAVAIRINST 4355.19DPolicy and process described in NAVAIRINST 4355.19D
Applies to Applies to all personnel supporting all NAVAIR and Aviation Program all personnel supporting all NAVAIR and Aviation Program pppp p pp g gp pp g gExecutive Officer (PEO) programs involved with the design, Executive Officer (PEO) programs involved with the design, development, test and evaluation, acquisition, indevelopment, test and evaluation, acquisition, in--service support, and service support, and disposal of naval aviation weapon systems and equipmentdisposal of naval aviation weapon systems and equipment
PhasesPhasesMaterial Solution AnalysisMaterial Solution Analysis11 –– ends with Milestone A ends with Milestone A (to (to begin technology begin technology developmentdevelopment))Technology DevelopmentTechnology Development –– ends with Milestone B ends with Milestone B (to (to begin SD&Dbegin SD&D))Engineering & Manufacturing DevelopmentEngineering & Manufacturing Development22 –– ends w/ Milestone C ends w/ Milestone C ((LRIPLRIP))P d ti & D l tP d ti & D l t d ith I iti l O ti C bilitd ith I iti l O ti C bilit ((IOCIOC))Production & DeploymentProduction & Deployment –– ends with Initial Operating Capability ends with Initial Operating Capability ((IOCIOC))Operations & SupportOperations & Support ((maintenancemaintenance))
11 Was previously called Concept RefinementWas previously called Concept Refinement22 W i l ll d S t D l t & D t tiW i l ll d S t D l t & D t ti
88
22 Was previously called System Development & DemonstrationWas previously called System Development & Demonstration
99
Found at: https://nserc.navy.mil/Pages/SETRTimeline.aspxhttps://nserc.navy.mil/Pages/SETRTimeline.aspxDocumented in: NAVAIRINST 4355.19D
Changes Based on new 5000.02Changes Based on new 5000.02IOCBA
MaterielSolution
FOCTechnology Production & Operations &
SEngineering and f
C
SolutionAnalysis
FRPDecisionReview
Materiel DevelopmentDecision
PDR CDR
CDD CPD
ICD
AoA
Post CDRAssessment
PDR
Development Deployment SupportManufacturing Development
Post PDRAssessment
Pre-Systems Acquisition Systems Acquisition Sustainmentor
Concept Refinement to Materiel Solution AnalysisConcept Refinement to Materiel Solution AnalysisConcept Refinement to Materiel Solution AnalysisConcept Refinement to Materiel Solution AnalysisSystem Development & Demonstration back to System Development & Demonstration back to Engineering & Manufacturing DevelopmentEngineering & Manufacturing DevelopmentThe Preliminary Design Review will be either before or The Preliminary Design Review will be either before or after Milestone B as defined in the program’s acquisition after Milestone B as defined in the program’s acquisition strategystrategy
1010
gygy
T P /Si G t PT P /Si G t PTwo Pass/Six Gate ProcessTwo Pass/Six Gate Process
Objective: Establish aObjective: Establish a disciplineddisciplined andand integratedintegratedObjective: Establish a Objective: Establish a disciplineddisciplined and and integratedintegratedprocess for requirements and acquisition decisionprocess for requirements and acquisition decision--making within DON, endorsing or approving keymaking within DON, endorsing or approving keymaking within DON, endorsing or approving key making within DON, endorsing or approving key Joint Capabilities Integration and Development Joint Capabilities Integration and Development Systems (JCIDS) and acquisition documents at Gate Systems (JCIDS) and acquisition documents at Gate reviews, and facilitating decisions regarding reviews, and facilitating decisions regarding required Navy and Marine Corps capabilities and required Navy and Marine Corps capabilities and acquisition of corresponding material solutionsacquisition of corresponding material solutionsacquisition of corresponding material solutions.acquisition of corresponding material solutions.
The SETR and the Two Pass/Six Gate process are complimentary
1111
The SETR and the Two Pass/Six Gate process are complimentary
Two Pass
Six Gate process
1212
Major Phases of SETRMajor Phases of SETR
1313
SETR Soft a e AspectsSETR Soft a e AspectsSETR Software AspectsSETR Software Aspects
Software is a key component in almost every Software is a key component in almost every SETRSETRThe NAVAIR Software Enterprise has developed The NAVAIR Software Enterprise has developed a set of templates for each reviewa set of templates for each review
Defines the contents of the software section for the Defines the contents of the software section for the reviewreviewIncludes a list of items to be covered along with theIncludes a list of items to be covered along with theIncludes a list of items to be covered along with the Includes a list of items to be covered along with the entrance criteria as it applies to softwareentrance criteria as it applies to software
1414
Example of Items from SRR TemplateExample of Items from SRR TemplateExample of Items from SRR TemplateExample of Items from SRR TemplateSW Development TeamSW Development TeamIntegrated Mater Schedule Highlighted with Software Integrated Mater Schedule Highlighted with Software MilMilMilestonesMilestonesSoftware Entrance CriteriaSoftware Entrance CriteriaRequirements Analysis and Allocation MethodologyRequirements Analysis and Allocation MethodologySystem Specifications TreeSystem Specifications TreeContract Data Requirements List (CDRL) Contract Data Requirements List (CDRL) Software Development Strategy Software Development Strategy Software Development ProcessSoftware Development ProcessSW Safety, Information Assurance and Security requirementsSW Safety, Information Assurance and Security requirementsSoftware Supplier ManagementSoftware Supplier ManagementSoftware MeasurementSoftware MeasurementSoftware Risk Assessment with Mitigation StrategiesSoftware Risk Assessment with Mitigation StrategiesIssues and ConcernsIssues and Concerns
1515
Systems Requirements ReviewSystems Requirements ReviewSystems Requirements ReviewSystems Requirements Review(SRR)(SRR)
1616
SRRSRRTechnical assessment establishing the system specification of Technical assessment establishing the system specification of the system under review to ensure a reasonable expectation the system under review to ensure a reasonable expectation of being judged operationally effective & suitable of being judged operationally effective & suitable Ensures the Capability Development Document (CDD), DOD Ensures the Capability Development Document (CDD), DOD Directives, statutory and regulatory guidance, and applicable Directives, statutory and regulatory guidance, and applicable public law has been correctly and completely represented in public law has been correctly and completely represented in the system specification and can be developed withinthe system specification and can be developed withinthe system specification and can be developed within the system specification and can be developed within program cost and schedule constraintsprogram cost and schedule constraints
Systems requirements are evaluated to determine whether they are fully Systems requirements are evaluated to determine whether they are fully defined and consistent with the mature system solution, and whether defined and consistent with the mature system solution, and whether y ,y ,traceability of systems requirements to the CDD is maintainedtraceability of systems requirements to the CDD is maintained
Assesses the performance requirements as captured in the Assesses the performance requirements as captured in the system specification, and ensures the preferred system system specification, and ensures the preferred system solution is:solution is:solution is:solution is:
Consistent with the system specificationConsistent with the system specificationCorrectly captures derived and correlated requirementsCorrectly captures derived and correlated requirementsHas well understood acceptance criteria and processesHas well understood acceptance criteria and processes
1717
Has well understood acceptance criteria and processesHas well understood acceptance criteria and processesIs achievable through available technologies resulting from the Technology Is achievable through available technologies resulting from the Technology Development phase Development phase
SRRSRR--I and SRRI and SRR--IIIISRRSRR I and SRRI and SRR IIIIFor major programs going through both concept For major programs going through both concept refinement/materiel solution analysis and technologyrefinement/materiel solution analysis and technologyrefinement/materiel solution analysis and technology refinement/materiel solution analysis and technology development, there will be 2 SRRsdevelopment, there will be 2 SRRsSRRSRR--II
Technical assessment establishing the specification of theTechnical assessment establishing the specification of theTechnical assessment establishing the specification of the Technical assessment establishing the specification of the system under review, previously represented by the system under review, previously represented by the Performance Based Specification (PBS), to continue the Performance Based Specification (PBS), to continue the requirements decomposition process prior to MS B requirements decomposition process prior to MS B
SRRSRR--IIIIThis review ensures the contractors participating in the This review ensures the contractors participating in the Technology Development (TD) Phase understands that the Technology Development (TD) Phase understands that the requirements of the contract including the system specificationrequirements of the contract including the system specificationrequirements of the contract including the system specification, requirements of the contract including the system specification, SOW, CDD, DoD Directives, statutory and regulatory guidance, SOW, CDD, DoD Directives, statutory and regulatory guidance, and applicable public law has been correctly and completely and applicable public law has been correctly and completely represented in the system specification and can be developed represented in the system specification and can be developed within program cost and schedule constraintswithin program cost and schedule constraints
1818
within program cost and schedule constraints within program cost and schedule constraints
SRR Soft a e P od ctsSRR Soft a e P od ctsSRR Software ProductsSRR Software ProductsSoftware Development Plan (SDP)Software Development Plan (SDP)
ScheduleScheduleScheduleScheduleProcessesProcessesSoftware Development EnvironmentSoftware Development Environment
System SpecificationSystem SpecificationI i i l S f R i D i i (SRD)I i i l S f R i D i i (SRD)Initial Software Requirements Description (SRD) Initial Software Requirements Description (SRD) Modeling and Simulation PlanModeling and Simulation PlanSupporting ProductsSupporting Products
Systems Engineering PlanSystems Engineering PlanSystems Engineering PlanSystems Engineering PlanRisk Management PlanRisk Management PlanMeasurement PlanMeasurement PlanQuality Assurance PlanQuality Assurance Plan
Measurement DataMeasurement DataCostCostSizeSizeRequirements VolatilityRequirements Volatility
1919
Requirements VolatilityRequirements Volatility
T h i l R di A tT h i l R di A tTechnical Readiness AssessmentTechnical Readiness Assessment(TRA)(TRA)( )( )
2020
TRATRATRATRA
A systematic metricsA systematic metrics based process thatbased process thatA systematic metricsA systematic metrics--based process that based process that assesses the maturity of Critical assesses the maturity of Critical Technology Elements (CTEs) by anTechnology Elements (CTEs) by anTechnology Elements (CTEs) by an Technology Elements (CTEs) by an independent panel of technical experts independent panel of technical experts Applicable to all acquisition categoryApplicable to all acquisition categoryApplicable to all acquisition category Applicable to all acquisition category (ACAT) programs per Secretary of the (ACAT) programs per Secretary of the Navy Instruction 5000.2CNavy Instruction 5000.2CyyMay be combined with a SRR May be combined with a SRR
2121
DefinitionsDefinitionsTRL Level Criteria
B i i i l b d d t d T iti f i tifi
Technology Readiness Level Definitions
1
Basic principles observed and reported: Transition from scientific research to applied research. Essential characteristics and behaviors of systems and architectures. Descriptive tools are mathematical formulations or algorithms.Technology concept and/or application formulated: Applied research Theory and scientific principles are focused on specific
2research. Theory and scientific principles are focused on specific application area to define the concept. Characteristicsof the application are described. Analytical tools are developed for simulation or analysis of the application.Analytical and experimental critical function and/or characteristic proof-of-concept: Proof of concept validation.
3
c a acte st c p oo o co cept oo o co cept a dat oActive Research and Development (R&D) is initiated with analytical and laboratory studies. Demonstration of technical feasibility using breadboard or brassboard implementations that are exercised with representative data.Component/subsystem validation in laboratory environment:
4 Standalone prototyping implementation and test. Integration of technology elements. Experiments with full-scale problems or data sets.
5
System/subsystem/component validation in relevant environment: Thorough testing of prototyping in representative
2222
5 environment. Basic technology elements integrated with reasonably realistic supporting elements. Prototyping implementations conform to target environment and interfaces.
Definitions cont.Definitions cont.TRL Level Criteria
System/subsystem model or prototyping demonstration in a
Technology Readiness Level Definitions continued
6
System/subsystem model or prototyping demonstration in a relevant end-to-end environment (ground or space): Prototyping implementations on full-scale realistic problems. Partially integrated with existing systems. Limited documentation available. Engineering feasibility fully demonstrated in actual system application.System prototyping demonstration in an operational
7
y p yp g penvironment (ground or space): System prototyping demonstration in operational environment. System is at or near scale of the operational system, with most functions available for demonstration and test. Well integrated with collateral and ancillary systems. Limited documentation available.
8
Actual system completed and "mission qualified" through test and demonstration in an operational environment (ground or space): End of system development. Fully integratedwith operational hardware and software systems. Most user d t ti t i i d t ti d i tdocumentation, training documentation, and maintenance documentation completed. All functionality tested in simulated and operational scenarios. Verification and Validation (V&V) completed.Actual system "mission proven" through successful mission operations (ground or space): Fully integrated with operational hardware/software systems Actual system has been thoroughly
23239
hardware/software systems. Actual system has been thoroughly demonstrated and tested in its operational environment. All documentation completed. Successful operational experience. Sustaining engineering support in place.
Integrated Baseline ReviewIntegrated Baseline ReviewIntegrated Baseline ReviewIntegrated Baseline Review(IBR)(IBR)
2424
IBRIBREmployed by Program Managers (PMs) throughout the Employed by Program Managers (PMs) throughout the life of projects where Earned Value Management (EVM) life of projects where Earned Value Management (EVM) i i di i dis required is required The IBR establishes a mutual understanding of the The IBR establishes a mutual understanding of the project Performance Management Baseline (PMB) and project Performance Management Baseline (PMB) and p o ides fo an ag eement on a plan of action top o ides fo an ag eement on a plan of action toprovides for an agreement on a plan of action to provides for an agreement on a plan of action to evaluate risks inherent in the PMB and the management evaluate risks inherent in the PMB and the management processes that operate during project execution processes that operate during project execution Assessment of risk within the PMB and the degree toAssessment of risk within the PMB and the degree toAssessment of risk within the PMB and the degree to Assessment of risk within the PMB and the degree to which the following have been established: which the following have been established:
Technical scope of workTechnical scope of workProject schedule key milestonesProject schedule key milestonesProject schedule key milestonesProject schedule key milestonesResourcesResourcesTaskTaskRationaleRationale
2525
RationaleRationaleManagement ProcessesManagement Processes
System Functional ReviewSystem Functional ReviewSystem Functional ReviewSystem Functional Review(SFR)(SFR)
2626
SFRSFRT h i l t t bli hi th t f ti l b li fT h i l t t bli hi th t f ti l b li fTechnical assessment establishing the system functional baseline of Technical assessment establishing the system functional baseline of the system under review to ensure a reasonable expectation of the system under review to ensure a reasonable expectation of being judged operationally effective and suitablebeing judged operationally effective and suitableAssesses the decomposition of the system specification to system Assesses the decomposition of the system specification to system p y p yp y p yfunctional specifications derived from use case analysisfunctional specifications derived from use case analysisFunctional requirements for operations and maintenance are Functional requirements for operations and maintenance are assigned to subassigned to sub--systems, hardware, software, or support after systems, hardware, software, or support after detailed reviews of the architecture in the environment it will bedetailed reviews of the architecture in the environment it will bedetailed reviews of the architecture in the environment it will be detailed reviews of the architecture in the environment it will be employed employed
The system’s lower level performance requirements are evaluated to The system’s lower level performance requirements are evaluated to determine whether they are fully defined and consistent with the determine whether they are fully defined and consistent with the mature system concept, and whether traceability of lowermature system concept, and whether traceability of lower--level systemslevel systemsmature system concept, and whether traceability of lowermature system concept, and whether traceability of lower level systems level systems requirements to toprequirements to top--level system performance and the CDD is level system performance and the CDD is maintained maintained
The development of representative operational use cases for the The development of representative operational use cases for the systemsystemsystemsystemThe SFR determines whether the systems functional definition is The SFR determines whether the systems functional definition is fully decomposed to its lower level, and that the Team is prepared fully decomposed to its lower level, and that the Team is prepared to start preliminary design for hardwareto start preliminary design for hardwareRi k M t M t Q lit A &Ri k M t M t Q lit A &
2727
Risk Management, Measurement, Quality Assurance, & Risk Management, Measurement, Quality Assurance, & Configuration Management processes fully functionalConfiguration Management processes fully functional
SFR ProductsSFR ProductsSFR ProductsSFR ProductsUpdates to SRR productsUpdates to SRR products
SDPSDPSystem SpecificationSystem SpecificationSystem Specification System Specification Modeling and Simulation PlanModeling and Simulation PlanSupporting ProductsSupporting Products
Systems Engineering PlanSystems Engineering PlanRi k M t PlRi k M t PlRisk Management PlanRisk Management PlanMeasurement PlanMeasurement PlanQuality Assurance PlanQuality Assurance PlanCost and Size EstimatesCost and Size Estimates
U d t d ( d t il) S ft R i t D i ti (SRD)U d t d ( d t il) S ft R i t D i ti (SRD)Updated (more detail) Software Requirements Description (SRD)Updated (more detail) Software Requirements Description (SRD)Interface Control DocumentsInterface Control DocumentsDraft Test PlanDraft Test PlanMeasurement DataMeasurement DataMeasurement DataMeasurement Data
CostCostSizeSizeRequirements VolatilityRequirements Volatility
2828
Software Specification ReviewSoftware Specification ReviewSoftware Specification ReviewSoftware Specification Review(SSR)(SSR)
2929
SSRSSRSSRSSRTechnical assessment establishing the software requirements Technical assessment establishing the software requirements baseline to ensure the preliminary design and ultimately the baseline to ensure the preliminary design and ultimately the software solution has a reasonable expectation of beingsoftware solution has a reasonable expectation of beingsoftware solution has a reasonable expectation of being software solution has a reasonable expectation of being judged operationally effective and suitablejudged operationally effective and suitable
The software’s lower level performance requirements are fully The software’s lower level performance requirements are fully defined and consistent with a mature system concept, and defined and consistent with a mature system concept, and
b l f lb l f l l l fl l f l ll ltraceability of lowertraceability of lower--level software requirements to toplevel software requirements to top--level system level system performance and the CDD is maintainedperformance and the CDD is maintained
A review of the finalized Computer Software Configuration A review of the finalized Computer Software Configuration Item (CSCI) requirements and operational conceptItem (CSCI) requirements and operational conceptItem (CSCI) requirements and operational conceptItem (CSCI) requirements and operational conceptSoftware Requirements Specification (Software Requirements Specification (SwRSSwRS) or Software ) or Software Requirements Description (SRD); Interface Requirements Requirements Description (SRD); Interface Requirements Specification(s) (IRS) or Software Interface RequirementsSpecification(s) (IRS) or Software Interface RequirementsSpecification(s) (IRS) or Software Interface Requirements Specification(s) (IRS) or Software Interface Requirements Description (Description (SIRDSIRD); Software Integration Plan; and the user’s ); Software Integration Plan; and the user’s Concept of Operation Description or User Documentation Concept of Operation Description or User Documentation Description Description form a satisfactory basis for proceeding into form a satisfactory basis for proceeding into p elimina soft a e designp elimina soft a e design
3030
preliminary software designpreliminary software design
SSR ProductsSSR ProductsSSR ProductsSSR ProductsUpdates to SRR and SFR productsUpdates to SRR and SFR productsFi l S ft R i t S ifi ti (S RS) S ftFi l S ft R i t S ifi ti (S RS) S ftFinal Software Requirements Specification (SwRS) or Software Final Software Requirements Specification (SwRS) or Software Requirements Description (SRD) Requirements Description (SRD) Requirements Verification MatrixRequirements Verification Matrix
CDD to specifications to SwRS or SRDCDD to specifications to SwRS or SRDCDD to specifications to SwRS or SRD CDD to specifications to SwRS or SRD Interface Control Documents (ICDs) and Interface Interface Control Documents (ICDs) and Interface Requirements Specification (IRS) or Software Interface Requirements Specification (IRS) or Software Interface Requirements description (SIRD) Requirements description (SIRD) q p ( )q p ( )Declassification, AntiDeclassification, Anti--Tamper, Open Architecture, and Tamper, Open Architecture, and Information Assurance requirementsInformation Assurance requirementsCompleted Test PlanCompleted Test PlanSoftware Integration PlanSoftware Integration PlanMeasurement DataMeasurement Data
CostCostSiSi
3131
SizeSizeRequirements VolatilityRequirements Volatility
Preliminary Design ReviewPreliminary Design ReviewPreliminary Design ReviewPreliminary Design Review(PDR)(PDR)
3232
PDRPDRTechnical assessment establishing the physically allocated Technical assessment establishing the physically allocated baseline to ensure that the system has a reasonable baseline to ensure that the system has a reasonable expectation of being judged operationally effective andexpectation of being judged operationally effective andexpectation of being judged operationally effective and expectation of being judged operationally effective and suitablesuitableAssesses the allocated designAssesses the allocated design captured in subsystem captured in subsystem product specifications for each configuration item in theproduct specifications for each configuration item in theproduct specifications for each configuration item in the product specifications for each configuration item in the system and ensures that each function, in the functional system and ensures that each function, in the functional baseline, has been allocated to one or more system baseline, has been allocated to one or more system configuration itemsconfiguration itemsSubsystem specifications for hardware and software, along Subsystem specifications for hardware and software, along with associated Interface Control Documents (ICDs), with associated Interface Control Documents (ICDs), enable detailed design or procurement of subsystems enable detailed design or procurement of subsystems A f l i i di t d th T ’A f l i i di t d th T ’A successful review is predicated on the Team’s A successful review is predicated on the Team’s determination that the determination that the subsystem requirements, subsystem subsystem requirements, subsystem preliminary design, results of peer reviews, and plans for preliminary design, results of peer reviews, and plans for development and testing form a satisfactory basis fordevelopment and testing form a satisfactory basis for
3333
development and testing form a satisfactory basis for development and testing form a satisfactory basis for proceeding into detailed design and test procedure proceeding into detailed design and test procedure developmentdevelopment
PDR ProductsPDR ProductsA
X Y
B
PDR ProductsPDR ProductsUpdates to SRR, SFR, and SSR productsUpdates to SRR, SFR, and SSR productsT L l S ft D i D i ti d/T L l S ft D i D i ti d/
X Y
Top Level Software Design Description and/or Top Level Software Design Description and/or Software Architecture Description Software Architecture Description Completed Test PlanCompleted Test PlanCompleted Test PlanCompleted Test PlanDraft Test ProceduresDraft Test ProceduresTraceability from design documentation to Traceability from design documentation to
b t t t i tb t t t i tsubsystem test requirements subsystem test requirements Representative mission profiles Representative mission profiles Measurement DataMeasurement DataMeasurement DataMeasurement Data
SizeSizeDefectsDefects
3434
Requirements VolatilityRequirements Volatility
Critical Design ReviewCritical Design ReviewCritical Design ReviewCritical Design Review(CDR)(CDR)
3535
CDRCDRTechnical assessment establishing the build baseline to Technical assessment establishing the build baseline to ensure that the system has a reasonable expectation of ensure that the system has a reasonable expectation of
ffffbeing judged operationally effective and suitable being judged operationally effective and suitable Assesses the final designAssesses the final design as captured in product as captured in product specifications for each configuration item in the system, specifications for each configuration item in the system, p g y ,p g y ,and ensures that item has been captured in detailed and ensures that item has been captured in detailed documentation documentation Product specifications for software enable coding of theProduct specifications for software enable coding of theProduct specifications for software enable coding of the Product specifications for software enable coding of the Computer Software Configuration Item (CSCI) Computer Software Configuration Item (CSCI) A successful review is predicated on the Team’s A successful review is predicated on the Team’s determination that thedetermination that the subsystem requirementssubsystem requirementsdetermination that the determination that the subsystem requirements, subsystem requirements, subsystem detail design, results of peer reviews, and subsystem detail design, results of peer reviews, and plans for testing form a satisfactory basis for proceeding plans for testing form a satisfactory basis for proceeding into implementation demonstration and testinto implementation demonstration and test
3636
into implementation, demonstration, and testinto implementation, demonstration, and test
CDR P od ctsCDR P od cts
class Graphical Displays
ValidationDisplay ConstraintFans RangeRings
«interface»IDrawable
CDR ProductsCDR ProductsUpdates to SRR SFR SSR & PDR productsUpdates to SRR SFR SSR & PDR products
DrawFactory
PayloadStatusIcon SafetyZones LaunchBasket
SafetyZoneParameterForm
Creates
Updates to SRR, SFR, SSR, & PDR productsUpdates to SRR, SFR, SSR, & PDR productsAll specifications and requirements documentation are All specifications and requirements documentation are complete/stablecomplete/stable
Detailed Software Design DescriptionDetailed Software Design DescriptionDetailed Software Design Description Detailed Software Design Description Units Test ProceduresUnits Test ProceduresTraceability Verification MatrixTraceability Verification Matrix
CDD t ifi ti t i t d t ti t d i tCDD t ifi ti t i t d t ti t d i tCDD to specifications to requirements documentation to design to CDD to specifications to requirements documentation to design to test procedurestest proceduresTraceability in both directionsTraceability in both directions
Measurement DataMeasurement DataMeasurement DataMeasurement DataSizeSizeDefectsDefectsMaturityMaturity
3737
MaturityMaturityRequirements VolatilityRequirements Volatility
Integration Readiness ReviewIntegration Readiness ReviewIntegration Readiness ReviewIntegration Readiness Review(IRR)(IRR)
3838
IRRIRRTechnical assessment establishing the configuration to be Technical assessment establishing the configuration to be used in integration test to ensure that the system has a used in integration test to ensure that the system has a reasonable expectation of being judged operationallyreasonable expectation of being judged operationallyreasonable expectation of being judged operationally reasonable expectation of being judged operationally effective and suitable effective and suitable A product and process assessment to ensure that A product and process assessment to ensure that hardware and software or software components are readyhardware and software or software components are readyhardware and software or software components are ready hardware and software or software components are ready to begin integrated configuration item (CI) testingto begin integrated configuration item (CI) testing
Assess prior component or unit level testing adequacy, test Assess prior component or unit level testing adequacy, test planning, test objectives, test methods and procedures, scope of planning, test objectives, test methods and procedures, scope of p g, j , p , pp g, j , p , ptests, and determines if required test resources have been tests, and determines if required test resources have been properly identified and coordinated to support planned testsproperly identified and coordinated to support planned testsVerifies the traceability of planned tests to program, engineering Verifies the traceability of planned tests to program, engineering data, analysis, and certification requirementsdata, analysis, and certification requirementsdata, analysis, and certification requirements data, analysis, and certification requirements
Testing is based upon the Test Plan (TP) Testing is based upon the Test Plan (TP) Begun in requirements phase and completed during designBegun in requirements phase and completed during design
Conducted after the test and/or validation procedures areConducted after the test and/or validation procedures are
3939
Conducted after the test and/or validation procedures are Conducted after the test and/or validation procedures are complete and unit level testing is completecomplete and unit level testing is complete
IRR P od ctsIRR P od ctsIRR ProductsIRR Products
Updates to SRR, SFR, SSR, PDR, & CDR Updates to SRR, SFR, SSR, PDR, & CDR productsproductsA d I i T PlA d I i T PlApproved Integration Test PlanApproved Integration Test PlanApproved Integration Test ProceduresApproved Integration Test ProceduresF t f I t ti T t R tF t f I t ti T t R tFormat for Integration Test ReportFormat for Integration Test ReportCompleted Integration Test Verification MatrixCompleted Integration Test Verification MatrixMeasurement DataMeasurement DataMeasurement DataMeasurement Data
Quality/maturityQuality/maturityDefectsDefects
4040
DefectsDefects
Test Readiness ReviewTest Readiness Review(TRR)(TRR)(TRR)(TRR)
4141
TRRTRRTechnical assessment establishing the configuration used in test Technical assessment establishing the configuration used in test to ensure that the system has a reasonable expectation of being to ensure that the system has a reasonable expectation of being judged operationally effective and suitablejudged operationally effective and suitablejudged operationally effective and suitable judged operationally effective and suitable TRR is a multiTRR is a multi--disciplined disciplined product and process assessment to product and process assessment to ensure that the subsystem, system, or systems of systems has ensure that the subsystem, system, or systems of systems has stabilized in configuration and is ready to proceed into formal stabilized in configuration and is ready to proceed into formal g y pg y ptesttest
Assesses prior unit level and system integration testing adequacy, test Assesses prior unit level and system integration testing adequacy, test planning, test objectives, test methods and procedures, scope of tests, and planning, test objectives, test methods and procedures, scope of tests, and determines if required test resources have been properly identified and determines if required test resources have been properly identified and q p p yq p p ycoordinated to support planned testscoordinated to support planned testsVerifies the traceability of planned tests to program, engineering, analysis, Verifies the traceability of planned tests to program, engineering, analysis, and certification requirementsand certification requirementsDetermines the completeness of test procedures and their compliance with Determines the completeness of test procedures and their compliance with p p pp p ptest plans and descriptionstest plans and descriptionsAssesses the impact of known discrepancies to determine if testing is Assesses the impact of known discrepancies to determine if testing is appropriate prior to implementation of the corrective fix appropriate prior to implementation of the corrective fix
TheThe TRR process is equally applicable to all tests in all phases ofTRR process is equally applicable to all tests in all phases of
4242
The The TRR process is equally applicable to all tests in all phases of TRR process is equally applicable to all tests in all phases of an acquisition programan acquisition program
TRR P od ctsTRR P od ctsTRR ProductsTRR Products
Updates to SRR, SFR, SSR, PDR, CDR, and IRR productsUpdates to SRR, SFR, SSR, PDR, CDR, and IRR productsTraceability AnalysisTraceability AnalysisApproved Test PlanApproved Test PlanApproved Test PlanApproved Test PlanApproved Test ProceduresApproved Test ProceduresFormat for Test ReportFormat for Test ReportppCompleted Test Verification MatrixCompleted Test Verification MatrixSoftware Version DocumentSoftware Version DocumentM DM DMeasurement DataMeasurement Data
Quality/maturityQuality/maturityDefectsDefects
4343
Flight Readiness ReviewFlight Readiness ReviewFlight Readiness ReviewFlight Readiness Review(FRR)(FRR)
4444
FRRFRRTechnical assessment establishing the configuration used in Technical assessment establishing the configuration used in flight test to ensure that the system has a reasonable flight test to ensure that the system has a reasonable expectation of being judged operationally effective andexpectation of being judged operationally effective andexpectation of being judged operationally effective and expectation of being judged operationally effective and suitablesuitableAssesses the system and test environment to ensure that Assesses the system and test environment to ensure that the system under review can proceed into flight testthe system under review can proceed into flight test with:with:the system under review can proceed into flight testthe system under review can proceed into flight test with:with:
NAVAIR airworthiness standards metNAVAIR airworthiness standards metObjectives clearly statedObjectives clearly statedFlight test data requirements clearly identifiedFlight test data requirements clearly identifiedFlight test data requirements clearly identifiedFlight test data requirements clearly identifiedAcceptable risk management plan defined and approved Acceptable risk management plan defined and approved
Ensures that:Ensures that:Proper coordination has occurred between engineering and flightProper coordination has occurred between engineering and flightProper coordination has occurred between engineering and flight Proper coordination has occurred between engineering and flight testtestAll applicable disciplines understand and concur with:All applicable disciplines understand and concur with:
The scope of effort that has been identifiedThe scope of effort that has been identifiedH thi ff t ill b t d t d i th d t (t ti fH thi ff t ill b t d t d i th d t (t ti f
4545
How this effort will be executed to derive the data necessary (to satisfy How this effort will be executed to derive the data necessary (to satisfy airworthiness and test and evaluation requirements) to ensure the airworthiness and test and evaluation requirements) to ensure the weapon system evaluated is ready to proceed to flight testweapon system evaluated is ready to proceed to flight test
FRR P od ctsFRR P od ctsFRR ProductsFRR Products
Updates to SRR, SFR, SSR, PDR, CDR, IRR, & Updates to SRR, SFR, SSR, PDR, CDR, IRR, & TRR productsTRR productsApproved Flight Test PlanApproved Flight Test PlanApproved Flight Test PlanApproved Flight Test PlanApproved Test Points/Flight Scenarios/Knee Approved Test Points/Flight Scenarios/Knee CardsCardsFormat for Flight Test ReportFormat for Flight Test ReportSoftware Version DocumentSoftware Version DocumentM t D tM t D tMeasurement DataMeasurement Data
Quality/maturityQuality/maturityDefects/Test HourDefects/Test Hour
4646
//
Operational Test Readiness ReviewOperational Test Readiness ReviewOperational Test Readiness ReviewOperational Test Readiness Review(OTRR)(OTRR)
4747
OTRROTRRMultiMulti--disciplined product and process assessment to disciplined product and process assessment to ensure that the system under review can proceed into ensure that the system under review can proceed into O ti l T t d E l ti (OT&E) ith hi hO ti l T t d E l ti (OT&E) ith hi hOperational Test and Evaluation (OT&E) with a high Operational Test and Evaluation (OT&E) with a high probability the system will successfully complete probability the system will successfully complete operational testing operational testing S ccessf l pe fo mance d ing Ope ational E al ationS ccessf l pe fo mance d ing Ope ational E al ationSuccessful performance during Operational Evaluation Successful performance during Operational Evaluation (OPEVAL) generally indicates the system being tested is (OPEVAL) generally indicates the system being tested is effective and suitable for Fleet introductioneffective and suitable for Fleet introduction
The decision to enter production may be based on this successfulThe decision to enter production may be based on this successfulThe decision to enter production may be based on this successful The decision to enter production may be based on this successful determination determination
Of critical importance to this review is the understanding Of critical importance to this review is the understanding ofof available system performance to meet the Capabilityavailable system performance to meet the Capabilityof of available system performance to meet the Capability available system performance to meet the Capability Production Document (CPD) Production Document (CPD) Operational requirements defined in the CPD must match Operational requirements defined in the CPD must match the Requirements tested to in the Test and Evaluation the Requirements tested to in the Test and Evaluation
4848
qqMaster Plan (TEMP) Master Plan (TEMP)
OTRR P od ctsOTRR P od ctsOTRR ProductsOTRR Products
Updates to SRR, SFR, SSR, PDR, CDR, IRR, TRR, Updates to SRR, SFR, SSR, PDR, CDR, IRR, TRR, & FRR products& FRR productsA d T & E l i M Pl (TEMP)A d T & E l i M Pl (TEMP)Approved Test & Evaluation Master Plan (TEMP)Approved Test & Evaluation Master Plan (TEMP)Approved Test Points/Flight ScenariosApproved Test Points/Flight ScenariosF t f O ti l T t R t & Q i kF t f O ti l T t R t & Q i k l kl kFormat for Operational Test Report & QuickFormat for Operational Test Report & Quick--look look ReportReportMeasurement DataMeasurement DataMeasurement DataMeasurement Data
Quality/maturityQuality/maturityDefects/Test HourDefects/Test Hour
4949
NAVAIRNAVAIR--Specific Lessons LearnedSpecific Lessons Learned
5050
Req i ements Lessons Lea nedReq i ements Lessons Lea nedRequirements Lessons LearnedRequirements Lessons LearnedSome developers tend to resist documentingSome developers tend to resist documentingSome developers tend to resist documenting Some developers tend to resist documenting requirements in a requirements traceability toolrequirements in a requirements traceability tool
Inability to trace requirements back to customer’s/sponsor’s Inability to trace requirements back to customer’s/sponsor’s requirementsrequirementsrequirementsrequirementsRequirements creep Requirements creep –– adding requirements not needed to meet adding requirements not needed to meet user’s/customer’s desiresuser’s/customer’s desires
Lack of concurrence among theLack of concurrence among the stakeholdersstakeholders of theof theLack of concurrence among the Lack of concurrence among the stakeholdersstakeholders of the of the requirements (collaboration)requirements (collaboration)
Key contributor to requirements instability, which leads to cost Key contributor to requirements instability, which leads to cost and schedule problemsand schedule problemspp
Requirements too loose/broadly written, making the Requirements too loose/broadly written, making the requirements decomposition more difficult requirements decomposition more difficult
5151
R i t L L d tR i t L L d tRequirements Lessons Learned cont.Requirements Lessons Learned cont.
Tendency to begin preliminary design beforeTendency to begin preliminary design beforeTendency to begin preliminary design before Tendency to begin preliminary design before requirements done or maturity of the requirements done or maturity of the requirements verifiedrequirements verified
C lt i l t f kC lt i l t f kCan result in a lot of reworkCan result in a lot of reworkIf the requirements are not fully defined, then your If the requirements are not fully defined, then your estimates for both cost and schedule will be estimates for both cost and schedule will be inaccurateinaccurateinaccurateinaccurate
Resistance to having a requirements change Resistance to having a requirements change control board early in the requirements phasecontrol board early in the requirements phasey q py q pLack of requirements stability measure (metric)Lack of requirements stability measure (metric)Requirements phase too little timeRequirements phase too little time
5252
Design Lessons Lea nedDesign Lessons Lea nedDesign Lessons LearnedDesign Lessons LearnedTendency to combine preliminary design and detailedTendency to combine preliminary design and detailedTendency to combine preliminary design and detailed Tendency to combine preliminary design and detailed design into a single phase to save timedesign into a single phase to save time
Peer reviews tend to have more defects because designs not Peer reviews tend to have more defects because designs not thought out wellthought out wellthought out wellthought out wellTendency to begin coding/production before design maturity Tendency to begin coding/production before design maturity verifiedverified
Those who use a design tool tend to do better designsThose who use a design tool tend to do better designsThose who use a design tool, tend to do better designs, Those who use a design tool, tend to do better designs, especially if requirements were managed with a toolespecially if requirements were managed with a toolSee some confusion between architecture definition and See some confusion between architecture definition and d id idesigndesign
These developers tend to begin coding early and call it These developers tend to begin coding early and call it prototypingprototyping
5353
Agile P og ammingAgile P og ammingAgile ProgrammingAgile Programming
Dilbert is copywrited by Scott Adams, Incorporated and is provided for non-commercial use by United Features Syndicate, Incorporated.
5454
Coding Lessons LearnedCoding Lessons LearnedCoding Lessons LearnedCoding Lessons LearnedSize tends to go up and amount of reuse tends to go down as a Size tends to go up and amount of reuse tends to go down as a function of timefunction of timefunction of timefunction of time
Add growth into planningAdd growth into planningTend to underestimate the amount of effort required for reusing Tend to underestimate the amount of effort required for reusing codecode
d f d ll b hd f d ll b hConsider if new code will be cheaperConsider if new code will be cheaperCan be as much as 1.5 times the cost of new codeCan be as much as 1.5 times the cost of new code
Auto code generator use increasingAuto code generator use increasingProblems/defects come from humansProblems/defects come from humansProblems/defects come from humansProblems/defects come from humansMost effective when used in conjunction with a design toolMost effective when used in conjunction with a design tool
Peer reviews are extremely important during this phasePeer reviews are extremely important during this phaseDetects problems when they are cheaper to fixDetects problems when they are cheaper to fix
Integrated unit testing tends to get shortened when schedules slideIntegrated unit testing tends to get shortened when schedules slideConsider sliding schedule or reducing content if schedule cannot slideConsider sliding schedule or reducing content if schedule cannot slide
Resource planning (labs, tools, and people) not necessarily well Resource planning (labs, tools, and people) not necessarily well thought out especially when there is a hardwarethought out especially when there is a hardware--inin--thethe--loop labloop lab
5555
thought out, especially when there is a hardwarethought out, especially when there is a hardware inin thethe loop labloop lab
Testing Lessons LearnedTesting Lessons LearnedggLate development of test plans and proceduresLate development of test plans and procedures
May not be able to get all the resources neededMay not be able to get all the resources neededMay not be able to get all the resources neededMay not be able to get all the resources neededMay not have the proper review or secured commitment to the May not have the proper review or secured commitment to the testingtesting
Too little time for testingToo little time for testingHave to rush through tests and may overlook problemsHave to rush through tests and may overlook problemsHave to cut out some of the testsHave to cut out some of the testsHave to test concurrent with development, which may cause Have to test concurrent with development, which may cause retest when tested areas of the software are changedretest when tested areas of the software are changedretest when tested areas of the software are changedretest when tested areas of the software are changed
Automated testing promotesAutomated testing promotesBetter coverage during testingBetter coverage during testingMore efficient use of staffMore efficient use of staffMore efficient use of staffMore efficient use of staffRepeatability of testingRepeatability of testingEfficient usage of test facilities during 2Efficient usage of test facilities during 2ndnd & 3& 3rdrd shiftsshifts
5656
Testing Lessons Learned cont.Testing Lessons Learned cont.Testing Lessons Learned cont.Testing Lessons Learned cont.Don’t document test sessionsDon’t document test sessions
Anomalies discovered in the test session may get overlookedAnomalies discovered in the test session may get overlookedAnomalies discovered in the test session may get overlookedAnomalies discovered in the test session may get overlookedDon’t have data to show what has been tested to support Don’t have data to show what has been tested to support decisionsdecisions
Readiness for next stepReadiness for next stepReadiness for next stepReadiness for next stepFlight clearancesFlight clearancesProduction decisionsProduction decisions
Can’t take credit for the testing performed and may have to redo Can’t take credit for the testing performed and may have to redo g p yg p ytestingtesting
Wrong configuration testedWrong configuration testedContention for test facilities due to poor upContention for test facilities due to poor up--frontfrontContention for test facilities due to poor upContention for test facilities due to poor up front front planningplanning
Schedule delaysSchedule delaysRemoval of testsRemoval of tests
5757
Removal of testsRemoval of tests
Schedule ComplianceSchedule Compliance
5858
Dilbert is copywrited by Scott Adams, Incorporated and is provided for non-commercial use by United Features Syndicate, Incorporated.
Schedule IssuesSchedule IssuesBeginning tasks before maturity or readiness has Beginning tasks before maturity or readiness has been verified rarely if ever saves time and been verified rarely if ever saves time and fundingfundingfundingfunding
JSFJSF11
The JSF started The JSF started system development before requisite system development before requisite technologies were readytechnologies were ready, , started manufacturing test started manufacturing test
i ft b f d i t bli ft b f d i t bl dd d td taircraft before designs were stableaircraft before designs were stable, and , and moved to moved to production before flight tests have adequately production before flight tests have adequately demonstrated that the aircraft design meets performance demonstrated that the aircraft design meets performance and operational suitability requirementsand operational suitability requirements. .
FCSFCS22FCSFCS22
The current FCS practice is to The current FCS practice is to overlap builds more than the overlap builds more than the traditional spiral model doestraditional spiral model does with software developers reporting with software developers reporting that that evolving requirements have caused them to interpret evolving requirements have caused them to interpret
d i l t h i i t ll i t thd i l t h i i t ll i t thand implement changes in requirements well into the and implement changes in requirements well into the design and code phasesdesign and code phases, compromising the amount of time , compromising the amount of time allotted for testingallotted for testing
1 Source: GAO1 Source: GAO--0808--388, 388, JOINT STRIKE FIGHTER: Recent Decisions by DOD Add to Program JOINT STRIKE FIGHTER: Recent Decisions by DOD Add to Program
5959
y gy gRisksRisks. March 2008 . March 2008 2 Source: GAO2 Source: GAO--0808--409, 409, DEFENSE ACQUISITIONS : Significant Challenges Ahead in DEFENSE ACQUISITIONS : Significant Challenges Ahead in Developing and Demonstrating Future Combat System’s Network and SoftwareDeveloping and Demonstrating Future Combat System’s Network and Software. March . March 20082008
Sched le D i en DemosSched le D i en DemosSchedule Driven DemosSchedule Driven Demos
Take your time and make sure the product is mature Take your time and make sure the product is mature b f i t th t tb f i t th t t
Dilbert is copywrited by Scott Adams, Incorporated and is provided for non-commercial use by United Features Syndicate, Incorporated.
6060
before going to the next stepbefore going to the next step
Schedule Lessons LearnedSchedule Lessons LearnedTendency for schedules to be too aggressive for the amount of Tendency for schedules to be too aggressive for the amount of work to be performedwork to be performed“When faced with unrealistic schedules, engineering teams often “When faced with unrealistic schedules, engineering teams often behave irrationally”*behave irrationally”*behave irrationally”*behave irrationally”*
Race through requirementsRace through requirementsProduce a superficial designProduce a superficial designRush into codingRush into codingS ft t d t b d li d l t th if it d l d ith ti lS ft t d t b d li d l t th if it d l d ith ti lSoftware tends to be delivered later than if it were developed with a rational Software tends to be delivered later than if it were developed with a rational planplan
Lack of staffing allocation across the scheduleLack of staffing allocation across the scheduleMost common problem is staff brought in too lateMost common problem is staff brought in too late
Overlapping of phases (to save time) and beginning next phase with anOverlapping of phases (to save time) and beginning next phase with anOverlapping of phases (to save time) and beginning next phase with an Overlapping of phases (to save time) and beginning next phase with an immature productimmature productCritical path is not evidentCritical path is not evident
Impacts risk managementImpacts risk managementP t i ti d ( h ti ) b did thi t iti tP t i ti d ( h ti ) b did thi t iti tPuts you in reaction mode (chaotic) because you did nothing to mitigate Puts you in reaction mode (chaotic) because you did nothing to mitigate
Insufficient detail to assess risksInsufficient detail to assess risksUnknown linkages and interdependencies of the tasks identifiedUnknown linkages and interdependencies of the tasks identified
6161
* Source: Humphrey, Watts S. “Winning in Software: An Executive Strategy.” Addison* Source: Humphrey, Watts S. “Winning in Software: An Executive Strategy.” Addison--Wesley, Wesley, 2002.2002.
Soft a e S pplie s LessonsSoft a e S pplie s LessonsSoftware Suppliers LessonsSoftware Suppliers LessonsIn an article in CrossTalk MagazineIn an article in CrossTalk Magazine** on software acquisition in the on software acquisition in the gg qqArmy, Edgar Dalrymple, the Program Manager for the Future Army, Edgar Dalrymple, the Program Manager for the Future Combat Systems Brigade Combat Team and Associate Director of Combat Systems Brigade Combat Team and Associate Director of Software and Distributed Systems, when answering a question Software and Distributed Systems, when answering a question making one change in the way the government procures software:making one change in the way the government procures software:g g y g pg g y g p
“The government, at least the Army, needs to stop buying “The government, at least the Army, needs to stop buying software exclusively from the traditional defense software exclusively from the traditional defense
i b Th i h h h di b Th i h h h dcontracting base. These companies have the overhead contracting base. These companies have the overhead costs of manufacturing companies, yet software costs of manufacturing companies, yet software development should carry a far smaller overhead development should carry a far smaller overhead burden Most defense contractors are still managed byburden Most defense contractors are still managed byburden. Most defense contractors are still managed by burden. Most defense contractors are still managed by manufacturing engineers or business managers.”manufacturing engineers or business managers.”
* Source: Starrett, Elizabeth. “Software Acquisition in the Army.” CrossTalk May 2007.* Source: Starrett, Elizabeth. “Software Acquisition in the Army.” CrossTalk May 2007.
6262
, q y y, q y y
S b t t M t LS b t t M t LSubcontractor Management LessonsSubcontractor Management Lessons
In major contracts where the prime has In major contracts where the prime has subcontracted out subsystems, we have seen:subcontracted out subsystems, we have seen:
S ft i t di t itS ft i t di t itSoftware requirements regarding process maturity Software requirements regarding process maturity and development of deliverables not passed down to and development of deliverables not passed down to the subcontractorthe subcontractorPrime subcontractor management personnel not Prime subcontractor management personnel not experienced in software developmentexperienced in software development
Results in latency of requirements changes to softwareResults in latency of requirements changes to softwarey q gy q gReworkReworkSchedule delaySchedule delay
Requirements volatilityRequirements volatility
6363
SummarySummarySummarySummaryThe Systems Engineering Technical Review The Systems Engineering Technical Review y g gy g gprocess is an iterative program timeline that process is an iterative program timeline that maps to DODI 5000.02 acquiaition timeline and maps to DODI 5000.02 acquiaition timeline and
bl hbl his compatible with:is compatible with:Basic Systems EngineeringBasic Systems EngineeringCMMICMMICMMICMMIIEEE 12207IEEE 12207
There are many software development lessonsThere are many software development lessonsThere are many software development lessons There are many software development lessons that NAVAIR has learned from its participation in that NAVAIR has learned from its participation in SETR eventsSETR events
6464
SETR events SETR events