james w. bilbro jb consulting international incose huntsville regional chapter monthly meeting...

JAMES W. BILBROJAMES W. BILBRO JB CONSULTING INTERNATIONAL JB CONSULTING INTERNATIONAL

INCOSE HUNTSVILLE REGIONAL CHAPTER MONTHLY MEETINGINCOSE HUNTSVILLE REGIONAL CHAPTER MONTHLY MEETING

THURSDAY, 18 THURSDAY, 18THTH, 2010, 2010

Benefits and Limitations of Current Benefits and Limitations of Current Techniques for Measuring the Readiness Techniques for Measuring the Readiness

of a System to Proceed to Productionof a System to Proceed to Production

2

Over the years, when things have dropped through the crack, processes have been “invented” to correct the problem - hence we have developed processes to identify:

•Technology Maturity•Manufacturing Maturity•Capability Maturity•System Maturity•Etc.

Technology Assessment vs. Technology Assessment vs. System AssessmentSystem Assessment

3

•All of these processes are in reality part of the systems engineering process, but because of what they deal with – e.g. “technology,” “manufacturing,” “etc.,” the processes are “owned” by organizations other than systems engineering.

•The result is a fragmented assessment process which in the end fails to achieve the desired goal of ensuring successful programs and projects.


4

•Everything that I am going to talk to you tonight falls under the purview of systems engineering!

•My focus will be on incorporating technology assessment into the system assessment process and then discussing other systems assessment process.

•Although I do not specifically address it, the same is true for Manufacturing Readiness Assessments.


5

Measuring the “readiness” of a system for production is essentially performing an assessment of the overall system with respect to identifying the risks remaining in the process of bringing it into operation.

Technology Assessment Technology Assessment vs. System Assessmentvs. System Assessment

6

The assessment should be performed not only with respect to the system itself, but also with respect to the interaction of the system with other systems within the overall System of Systems in which it is to operate!


7

A “system” assessment must have technology assessment as a component if it is to be successful in identifying the risks associated with bringing the system to fruition.


8

Technology is probably not what you think it is!


9

What is Technology?

“1.a. The application of science, especially to industrial or commercial* objectives. b. The entire body of methods and materials used to achieve such objectives.”

- The American Heritage Dictionary

* Or DOD or NASA


10

What is Technology?

•In general, technology development lies within the context of part “a” of the definition.

•However, the engineering community often makes use of the term “technology” within the context of part “b”. In this context, technology may be “old (passe),” “off-the-shelf (commercially available),” or new.

•It is further complicated when “old” technology, i.e. “heritage elements” are used in completely new ways – a situation that encompasses both parts “a” & “b”.


11

What is Technology?

•Using “heritage” or “legacy” systems, sub-systems or components (whether hardware or software) merits a more detailed discussion.•When “heritage” or “legacy” elements are used, it is often thought that because they are “mature” i.e. they have been in “operation” then we can just plug them in.•As a result, the systems engineering associated with that element is short circuited and the result is frequently disastrous.


12

What is Technology?

•The reason that use of “heritage” or “legacy” system is so often fraught with danger is that it is being asked to operate in an architecture or an environment that in all probability is different from that for which it was designed.

•And because we have not done the upfront analysis, we don’t know we are in trouble until we are already there!

•This has prompted a different definition of technology based on the experience of the developers.


13

•Engineering development is based on the premise that:

•Requirements are achievable and that any engineering difficulty can be overcome and the requisite performance achieved within cost and schedule constraints.

•The “certainty” associated with engineering development lies in the past experience of the developers.


14

•Technology development, however is based on the premise that:

•Requirements may not be achievable and that it is not known with any certainty whether or not they can be met within a given cost and schedule.

•Technology development, therefore, is all about maximizing the probability of success.

•Pursuing parallel paths.•Having alternate solutions.•Having fall-back positions.


15

Comparing the engineering development process to the technology development process one can see that it is extremely important to know in what type of development one is engaged.


16

Technology development is distinguished from engineering development in that it requires venturing into the realm of unknowns - beyond the ability of individuals to make informed judgments based on their experience.


17

Within the context of the expanded definition of technology we see that when using “heritage” or “legacy” systems we can easily find ourselves in a technology development effort whether we want to be or not!


18

So - what does Technology Impact?So - what does Technology Impact?

All aspects of the Systems Engineering Process!All aspects of the Systems Engineering Process!

Stakeholder Expectation:Stakeholder Expectation: Requirements Definition:Requirements Definition: Design Solution: Design Solution: Risk Management:Risk Management: Technical Assessment:Technical Assessment: Trade Studies:Trade Studies: Verification/Validation: Verification/Validation: Lessons Learned:Lessons Learned:


19

Consequently, for our first System Assessment, let’s Consequently, for our first System Assessment, let’s perform a technology assessment – but from a total perform a technology assessment – but from a total system perspective that includes its interaction with system perspective that includes its interaction with other systems in the system of systems in which it other systems in the system of systems in which it is to operate.is to operate.


20

A Technology Readiness Level (TRL), describes the maturity of a A Technology Readiness Level (TRL), describes the maturity of a given technology relative to its development cycle. given technology relative to its development cycle.

At its most basic, it is defined at a given point in time by what has At its most basic, it is defined at a given point in time by what has been done and under what conditions. been done and under what conditions.

Technology Readiness Level Technology Readiness Level (TRL)(TRL)

But – while a system/technology assessment But – while a system/technology assessment may determine the current maturity of a may determine the current maturity of a system it does nothing to inform the program system it does nothing to inform the program of what is required to successfully complete of what is required to successfully complete the development process.the development process.


Advancement Degree of Advancement Degree of Difficulty (ADDifficulty (AD22))

Advancement Degree of Difficulty (AD2) is a method of systematically Advancement Degree of Difficulty (AD2) is a method of systematically dealing with aspects beyond TRL.dealing with aspects beyond TRL.

It is a “predictive” description of what is required to move a system, It is a “predictive” description of what is required to move a system, subsystem or component from one TRL to another.subsystem or component from one TRL to another.

It provides information in the form of:It provides information in the form of:Likelihood of occurrence of an adverse event. Likelihood of occurrence of an adverse event. RiskRiskCost to ensure that such an event does not occur.Cost to ensure that such an event does not occur.The time required to implement the necessary action.The time required to implement the necessary action.

AD2 consists of a set of questions in 5 specific areas:Design and AnalysisManufacturingSoftware DevelopmentTestOperations

Impact

The levels of risk associated with AD2 are described in terms of the experience base of the developers.

i.e., have they done this before?

AD2 Tool Question Set AD2 Tool Output

Advancement Degree of Advancement Degree of Difficulty (ADDifficulty (AD22))

Risk Identification, Risk Identification, Integration & Illities (RI3)Integration & Illities (RI3)

RI3 used to support, not replace, existing Risk Identification process

Questions in nine ‘ilities areasDesign Maturity and StabilityScalability & ComplexityIntegrabilityTestabilitySoftwareReliability MaintainabilityHuman factorsPeople, organization, & skills

RI3 is a methodology for identifying frequently occurring risks by exploiting “lessons learned” and “best practice” based on case studies and the experience of the Air Force-wide development team.

Questions based on commonly occurring problems are contained in a guidebook and an Excel tool - a web based tool is under development.

0 1 2 3 4 5

Integrability

Testability

Reliability

Maintainability

Human Factors

Scalability & Complexity

People, Organization, Skills

Design Maturity & Stability

Consequence

Questions:• Integration• ilities

Risks

Additional Summary Displays

Risk Management

Step 2.Risk

Identification

Active Risk Manager

(ARM) compatible

file

Similar output for cost estimation

being investigated

Tool

Tool

PoPS*

Tool

RI3Guidebook

Risk Identification, Risk Identification, Integration & Illities (RI3)Integration & Illities (RI3)

Least Pressing Most Pressing

*Probability of Program Success

Other System Assessment Processes

System AssessmentSystem Assessment

SRLs are an analysis of key outputs of an acquisition project structured in such a way as to provide an understanding of work required to mature the project.

The SRL analysis is achieved using a matrix to capture the results of a comprehensive set of questions centered around System Engineering Drivers (SEDs) and selected systems disciplines (i.e., Training, Safety and Environment, etc.) and understand how they should mature over time.

The SRL analysis employs TRL analyses to provide a means of progressively measuring project maturity at technology, component, sub system and whole system levels.TRLsystem < TRLcomponent

System Readiness Level (SRL) System Readiness Level (SRL) – UK Ministry of Defense– UK Ministry of Defense

N.B. – Integration Readiness Levels (IRLs) & Design Readiness Levels (DRLs) were initially used but later rejected.

System Readiness Level (SRL) System Readiness Level (SRL) – UK Ministry of Defense– UK Ministry of Defense

SRLs are intended to be ‘descriptive’ and not ‘absolute’ as work on each systems discipline may progress at different rates.

An SRL assessment therefore produces a ‘signature’ rather than an absolute single point SRL figure.

The signature records the variation of maturity that has been achieved across the systems disciplines, acknowledging that not all projects mature against the systems disciplines at a consistent rate. The color of the boxes in the Systems Maturity Matrix is determined by analysis of the SRL signature obtained against the expectations for SRL maturity at the time of review

SRL Self Assessment Tool Results

DOD Systems Engineering DOD Systems Engineering ChecklistsChecklists

DOD Systems Engineering Checklists:

There are 18 System Engineering Checklists covering all program phases intended to supplement the Services individual processes/methodologies.

•e.g. The TRA checklist for consists of 69 questions in 8 areas: Timing/Entry Level, Planning, Program Schedule, Program Risk Assessment, Critical Technologies Identification, TRA Panel, TRA Preparation and Event and Completion/Exit Criteria. • Each question is to be assessed with respect to risk categories of Red, Yellow, Green, Unassigned or Not Applicable.

System Readiness Level System Readiness Level (SRL) – the Stevens Institute(SRL) – the Stevens Institute

The SRL in this case is defined through the combination of the TRL of a given technology with the Integration Readiness Level (IRL) of each of the elements with which it will be integrated.

SRLi = f(TRLj, IRLij)

Integration Readiness Levels

The overall SRL will be a function of the individual subsystem SRLi

SRL = f(SRL1, SRL2, …SRLn)

System Readiness Level System Readiness Level (SRL) – the Stevens Institute(SRL) – the Stevens Institute

The computation of SRL is considered as a normalized matrix of pairwise comparisons of normalized TRL and IRL.

System Maturity Optimization is underway at Stevens

Additional Areas that have Additional Areas that have been addressed with varying been addressed with varying

degrees of successdegrees of success

Design Readiness Level (DRL)Manufacturing Readiness Level (MRL)Integration Readiness Level (IRL)Software Readiness Level (SRL)Operational Readiness Level (ORL)Human Readiness Levels (HRL)Capability Readiness Level (CRL)Organizational Readiness Level(ORL)Programmatic Readiness Level(PRL

33

Technology Assessment is a vital part of any Technology Assessment is a vital part of any overall system maturity assessment.overall system maturity assessment.

There are many approaches to overall There are many approaches to overall system assessment.system assessment.

Any successful approach for system Any successful approach for system maturity assessment must balance the need maturity assessment must balance the need for data against the resources required to for data against the resources required to obtain that data.obtain that data.

Summary

34

Sadin, Stanley T.; Povinelli, Frederick P.; Rosen, Robert, “NASA Sadin, Stanley T.; Povinelli, Frederick P.; Rosen, Robert, “NASA technology push towards future space mission systems,” Space technology push towards future space mission systems,” Space and Humanity Conference Bangalore, India, Selected and Humanity Conference Bangalore, India, Selected Proceedings of the 39th International Astronautical Federation Proceedings of the 39th International Astronautical Federation Congress, Acta Astronautica, pp 73-77, V 20, 1989Congress, Acta Astronautica, pp 73-77, V 20, 1989

Mankins, John C. “Technology Readiness Levels” a White Mankins, John C. “Technology Readiness Levels” a White Paper, April 6, 1995.Paper, April 6, 1995.

Nolte, William, “Technology Readiness Level Calculator, Nolte, William, “Technology Readiness Level Calculator, “Technology Readiness and Development Seminar, “Technology Readiness and Development Seminar, Space Space System Engineering and Acquisition Excellence ForumSystem Engineering and Acquisition Excellence Forum, The , The Aerospace Corporation, April 28, 2005.Aerospace Corporation, April 28, 2005.

Mankins, John C. , “Research & Development Degree of Mankins, John C. , “Research & Development Degree of Difficulty (RD3)” A White Paper, March 10, 1998.Difficulty (RD3)” A White Paper, March 10, 1998.

BibliographyBibliography

35

Ramirez-Marquez, J.E. Sauser, B.J. “System Development Ramirez-Marquez, J.E. Sauser, B.J. “System Development Planning via System Maturity Optimization,” Accepted for future Planning via System Maturity Optimization,” Accepted for future publication in publication in IEEE Transactions on Engineering IEEE Transactions on Engineering Management, Management, IEEExplore.IEEExplore.

Bilbro, James W. “Bilbro, James W. “Systematic Assessment of the Program/Project Impacts of Technological Advancement and Insertion Revision A,” http://www.jbconsultinginternational.com


36

TOOLSTOOLS

RI3 Tool and Guidebook are available at: RI3 Tool and Guidebook are available at: http://www.afit.edu/cse/page.cfm?http://www.afit.edu/cse/page.cfm?page=164&sub=95page=164&sub=95

AD2 Tool along with integrated TRL tool available at: AD2 Tool along with integrated TRL tool available at: http://www.jbconsultinginternational.comhttp://www.jbconsultinginternational.com TRL Calculator is available at Website at: TRL Calculator is available at Website at:

https://acc.dau.mil/communitybrowser.aspx?id=25811https://acc.dau.mil/communitybrowser.aspx?id=25811 UK MOD Tool is available at: UK MOD Tool is available at:

http://www.aof.mod.uk/aofcontent/tactical/techman/index.htmhttp://www.aof.mod.uk/aofcontent/tactical/techman/index.htm Stevens SRL Tool is under development at: Stevens SRL Tool is under development at:

http://www.systemreadinesslevel.com/http://www.systemreadinesslevel.com/ Manufacturing Readiness Level Tool is available at: Manufacturing Readiness Level Tool is available at:

https://acc.dau.mil/CommunityBrowser.aspx?id=18231https://acc.dau.mil/CommunityBrowser.aspx?id=18231 DOD Engineering Checklists are available at:DOD Engineering Checklists are available at:https://acc.dau.mil/CommunityBrowser.aspx?id=144143&lang=en-UShttps://acc.dau.mil/CommunityBrowser.aspx?id=144143&lang=en-US


james w. bilbro jb consulting international incose huntsville regional chapter monthly meeting...

Documents

system assessment slide

nasa technology assessment

system assessment process

systems assessment process

old technology

term technology

technology development

fragmented assessment