integration-centric approach to system readiness assessment based on evidential reasoning

Journal of Systems Engineering and Electronics

Vol. 23, No. 6, December 2012, pp.881–890

Integration-centric approach to system readinessassessment based on evidential reasoning

Leilei Chang, Mengjun Li*, Ben Cheng, and Ping Zeng

School of Information System and Management, National University of Defense Technology,Changsha 410073, P. R. China

Abstract: An integration-centric approach is proposed to han-dle inadequate information in the system readiness level (SRL)assessment using the evidential reasoning (ER) algorithm. Cur-rent SRL assessment approaches cannot be applied to handleinadequate information as the input. The ER-based approachis proposed to synthesize inadequate input information and anintegration-centric perspective is applied to reduce the computa-tional complexity. Two case studies are performed to validate theefficiency of the proposed approach. And these studies are alsoperformed to study how the inadequate information will affect theassessment result. And the differences caused by the system’sstructure. The importance of the system’s structure in the SRLassessment is demonstrated and the contributions made in thisstudy are summarized as conclusions.

Keywords: system readiness level (SRL), technology readinesslevel (TRL), integration readiness level (IRL), evidential reasoning(ER), integration-centric.

DOI: 10.1109/JSEE.2012.00108

1. Introduction

The development of a new system depends on the priorsuccess of advanced technologies, thus it is important toperform technology assessment when starting a system[1]. Therefore, the concept of the technology readinesslevel (TRL) was introduced by NASA to communicatethe status of new technologies [2]. However, TRL wasdesigned for single technologies and it cannot satisfy thedemand for a comprehensive understanding of a system[3]. The concept of the system readiness level (SRL) wasintroduced by [3–7] to meet this challenge. An approachwas also proposed to calculate the SRL with the TRL andintegration readiness level (IRL) [6,7], a metric to eval-uate the integration readiness between two technologies,

Manuscript received November 21, 2011.*Corresponding author.This work was supported by the National Natural Science Foundation

of China (70901074; 71001104).

were combined.

References [3,6,7] assumed that TRL and IRL wereunique determined values, which indicated that the inputinformation were certain and adequate. Reference [8,9]extended SRL assessment by assuming that the values ofTRL and IRL followed a probabilistic distribution and thuscould handle the uncertain and adequate information inSRL assessment. Therefore, current SRL assessment ap-proaches can be applied when the input information is ad-equate, either being unique deterministic values or multi-value under the uncertainty.

However, the TRL and IRL scales are mainly derivedfrom experts’ knowledge and experience. Once humanknowledge is involved, it is inevitable to address an uncer-tainty caused by the vagueness intrinsic in human know-ledge and inadequacy caused by the limit of human know-ledge [10]. Therefore there is still a challenge in currentSRL assessment: the inability to handle inadequate infor-mation under the uncertainty, which is the motive of thisstudy.

In this study, an evidential reasoning (ER)-based ap-proach is proposed to handle the inadequate input infor-mation in the SRL assessment. The ER algorithm [11,12] is based on the multi-attribute assessment frameworkand D-S evidence theory [13] and has an instinctive in-sight in dealing with the uncertain and inadequate infor-mation [11,12]. In recent years, ER has been applied tothe decisionmaking, risk assessment, organizational self-evaluation, and supply chain problems in the engineeringdesign [14−17]. An integration-centric perspective is ap-plied in the ER-based approach, which consists of twoparts: first calculating the “component SRL” of each in-tegration, denoted as the “ISRL”, and then synthesizingthe “ISRLs” into the system’s SRL.

Two cases, each under multiple conditions, are studiedto validate the effectiveness of the proposed approach and

882 Journal of Systems Engineering and Electronics Vol. 23, No. 6, Dcemeber 2012

to reveal the differences caused by the structure of the sys-tem. This study is concluded by summarizing the contri-bution of this study and delineating the importance of thesystem’s structure in SRL assessments.

2. TRL, IRL and SRL

In this section, the concepts of TRL, IRL and SRL will bebriefly introduced in a general sense, and the challenge ofcurrent SRL assessment approaches will be discussed.

2.1 TRL and IRL

The term “TRL” was first introduced with seven levels byNASA in 1989 [18]. In 1995, NASA issued a white paper[1], in which TRL was extended to nine levels. Since theintroduction of TRL, it has become a widely accepted andmost prevailing tool in analyzing technical risk and techni-cal evaluation.

However, several challenges regarding TRL have beenraised as well [1,3,19]. Firstly, TRL neglects the integra-tion between technologies. Secondly, TRL does not sup-port the analysis of the uncertainty that may be caused byhuman or technical factors. Finally, TRL does not supporta comparative analysis when several technologies are in-volved.

Reference [3] concluded that the main limitation of TRLis that it is designed for a single technology. Thus, it cannotsatisfy the demands of a comprehensive understanding ofa system. This conclusion is in agreement with this study.The limitation of TRL leads to the introduction of the con-cept of IRL.

IRL was first proposed by [6,7] with nine levels, forthe following four purposes: (i) An IRL provides a metricthat can be understood by all of the relevant stakeholders.(ii) An IRL can evaluate the integration readiness betweentwo technologies. (iii) An IRL can be used with a TRLto determine a system readiness. (iv) An IRL providesa direction for improving the integration with other tech-nologies. Detailed definitions of TRL/IRL can be found in[2,6,7].

2.2 Current SRL assessment

2.2.1 SRL by Sauser

SRL is defined by Sauser as “an index of maturity from 0to 1 applied at the system-level with the objective of cor-relating this indexing to appropriate systems engineeringmanagement principles” [4].

Steps of Sauser’s SRL assessments are given asfollows [5]:

Step 1 TRL is defined in (1) as a vector, whereTRLi is the TRL of the ith technology and thevalue of TRLi is normalized to be between 0 and 1.

[TRL]n×n = [ TRL1 TRL2 · · · TRLn ]Tnorm.

(1)Step 2 IRL is defined in (2) as a matrix, where IRLij

is the IRL between the ith technology and the jth technol-ogy and the value of IRLij is normalized to be between 0and 1.

[IRL]n×n =

⎡⎢⎢⎢⎣IRL11 IRL12 · · · IRL1n

IRL21 IRL22 · · · IRL2n

......

. . ....

IRLn1 IRLn2 · · · IRLnn

⎤⎥⎥⎥⎦norm

.

(2)Note that IRLii denotes the hypothetical integration of

the ith technology to itself and IRLii = 1.

Step 3 Calculate the “component SRL” of each tech-nology, denoted as “SRLi” of the ith technology, whenintegrated into the system. The calculation of SRLi isgiven by

[SRL1 SRL2 · · · SRLn ]T =

[IRL]n×n × [TRL]n×1. (3)

Note that the value of SRLi of the ith technologywould fall within the interval (0, n) since every integra-tion of the ith technology, with every technology includingwith itself, is calculated.

Step 4 Calculate SRL of the system. The SRL of thesystem is the average of all “component SRL” of each tech-nology. The calculation of SRL is given by

SRL =SRL1/n1 + SRL2/n2 + · · · + SRLn/nn

n(4)

where ni is the number of integrations of technology i plusthe integration to itself, and n is the number of technolo-gies of the system.

Sauser’s SRL assessment can be divided into two parts.The first three steps (Steps 1−3) are the first part, the pur-pose of which is to obtain the “component SRL” of eachtechnology. The fourth step (Step 4) as the second part,is to obtain the composite SRL of the system. It can beconcluded that Sauser’s SRL assessment mainly focuseson the role that the technology plays; in other words, it isthe “technology-centric”.

SRL consists of five levels and can measure where a sys-tem is in its life cycle. SRL and its definitions are providedas in Table 1.

Table 1 SRL and its definitions

SRL Definition0.90 1.00 Operation and support0.80 0.89 Production and deployment0.60 0.79 Engineering and manufacturing development0.40 0.59 Technology development0.10 0.39 Concept refinement

Leilei Chang et al.: Integration-centric approach to system readiness assessment based on evidential reasoning 883

2.2.2 SRL by Tan

Since the values of TRL and/or IRL are derived from ex-perts, the ambiguity in the information cannot be omittedand there must be uncertain information in TRL and/or IRLassessment. This challenge is acknowledged by [3] as well.Reference [20] also pointed out the necessity to place aconfidence level on the information of the estimated value.

To extend Sauser’s SRL assessment to a more versatilecondition, [8, 9] proposed an approach using Monte Carlosimulation to assess SRL. In Tan’s SRL assessment, thevalues of TRL and/or IRL are assigned with a probabilisticdistribution. For example, the value of a technology’s TRLmay be assessed as 6, 7 and 8 with the possibilities of 0.3,0.5 and 0.2. Then a Monte Carlo simulation is performed.For each iteration, the value of each TRL/IRL is determin-istic, single-valued and is chosen by its possibility. There-fore, Sauser’s approach is still applicable to calculate the“component SRL” for each iteration. After a certain num-ber of iterations, the component SRL of each technologyis synthesized into an overall SRL of the system using anapproach proposed by Tan. The final result is presented intwo forms, the mean value and the a confidence interval.

2.3 Challenge of the current SRL assessment

approaches

Another limitation of the human judgment is that the hu-man being can only hold ultimate responsibilities undermost conditions and their personal experience, knowledgeand preference plays an irreplaceable role in making finaldecisions. Thus when the human knowledge is adopted, itis inevitable to address the inadequacy caused by the limi-tations of human knowledge [10].

Since Tan has proposed an approach that handles theuncertain but still adequate information, a simple yet nat-ural idea is to further extend Tan’s approach to handle theinadequate information. We would like to address the in-feasibility of this idea in the following discussion, in whichthe inadequacy is denoted as “N/A” and so is the same asin the rest of this study.

Following Tan’s SRL assessment, the TRL of a tech-nology may be 6, 7, or 8 with probabilities of p1, p2, p3

and p1 + p2 + p3 = 1. Add the “N/A” to Tan’s SRL as-sessment, then the TRL of a technology may be 6, 7, 8,or N/A with probabilities of p1, p2, p3, p4. Additionallyp1 + p2 + p3 + p4 = 1. For the next step of the simulation,to calculate the SRL for each iteration, each of the poten-tial value of the TRL is chosen based on its probabilities. IfN/A is chosen with a probability of p4, then the calculationprocess cannot go further because Tan adapted Sauser’ ap-proach and Sauser’s approach cannot handle “N/A”. Thus,

there is a dilemma because the inadequate information iseasy to introduce, identify and transform, but it cannot beprocessed.

Due to the inability of handling the inadequate infor-mation, an integration-centric, ER-based approach is pro-posed.

3. ER based SRL assessment

As discussed in Section 2.2.1 that the SRL assessment bySauser can be divided into two parts, first obtaining thecomponent SRL of each technology, and then synthesizingall of the component SRLs into system’s SRL.

Similarly, our approach is divided into two parts as well,but from an “integration-centric” perspective. Firstly, thecomponent SRL for the integration between any two tech-nologies, denoted as “ISRL”, is calculated. Then theISRLs are synthesized into the system’s SRL. Note thatthe “component SRL” in the approach is the SRL of the in-tegration, which is different from the “component SRL ofa technology” in Sauser’s SRL assessment approach.

For the specific algorithm to be applied, the followingtakes places: in the first part, Sauser’s approach is used toderive the ISRL for each possible combination of TRLand IRL and the ER algorithm is applied to synthesize theISRL for each combination into the ISRL of the integra-tion. In the second part, the ER algorithm is applied for asecond time to synthesize all of the ISRLs of the integra-tion into the SRL of the system.

3.1 Model assumption

3.1.1 Belief structure for TRL and IRL

The belief structure [12] is applied to demonstrate the un-certainty and inadequacy of the input data and assessmentresult.

Suppose that an assessment on the object O is per-formed and S(O) is the assessment result. Thereare N kinds of S(O), denoted as {G1, G2, . . . , GN}.The beliefs of reaching the N kinds of S(O)are {β1,O, β2,O, . . . , βN,O} and therefore S(O) ={{G1, β1,O}, {G2, β2,O}, . . . , {GN , βN,O}}, here βk,o �

0,N∑

k=1

βk,o � 1 (k = 1, 2, . . .N).

βk,o = 1 means that the assessment result is a certain

value, which suggests that S(O) = G. IfN∑

k=1

βk,o = 1,

then the assessment result information is adequate. IfN∑

k=1

βk,o < 1, then the assessment result information is

inadequate.


3.1.2 Integration-centric perspective

The integration-centric perspective provides a differentpoint of view when attempting to understand a system, be-cause the focus is transferred from the technology to the in-tegration between two technologies. This perspective alsobrings convenience for calculation by reducing the compu-tational complexity, which is illustrated as follows: Firstly,a system has n technologies and any two technologies havean integration. Secondly, the TRL for any technology has10 potential values. Finally, the IRL for any integration has10 potential values.

From a “technology-centric” perspective, to calculatethe component SRL for a combination of any technologieswould result in 10n × 10n−1 possible combinations, be-cause there would be n technologies and n−1 integrationsinvolved (the integration with itself is “9”).

From an “integration-centric” perspective, to calculatethe component SRL for a combination of any integration,the ISRL would result in 103 possible combinations, be-cause there would be two technologies and one integrationinvolved.

This simple scenario demonstrates that the calculationcomplexity decreases to a determined number and thusmakes the proposed approach valid and feasible in thecomputation.

3.2 The calculation of ISRL

Suppose that a system is composed of n technologies andthere is an integration between any two technologies. Forany integration, there must be and can only be two tech-nologies involved, which means that there are only threevariables in the calculation of one integration.

The ISRL of the integration between the ith techno-logy and the jth technology is defined as “ISRLij”. Al-though it is novel that there are nine scales for TRL andIRL, the background is still set to reflect a general perspec-tive. This scenario is defined as follows:

Defination 1 There are L1 alternatives (“N/A” notincluded) of TRL for the ith technology, denoted asTRLi,l1 , and the belief of reaching TRLi,l1 is εi,l1 . TheTRL for the ith technology is {(TRLi,l1, εi,l1)}, wherei = 1, 2, . . . , n and l1 = 1, 2, . . . , L1.

Defination 2 There are L2 alternatives (“N/A” notincluded) of TRL for the jth technology, denoted asTRLj,l2 , and the belief of reaching TRLj,l2 is εj,l2 . TheTRL for the jth technology is {(TRLj,l2, εj,l2)}, wherej = 1, 2, . . . , n and l2 = 1, 2, . . . , L2.

Defination 3 There are L3 alternatives (“N/A” not in-cluded) of IRL for the integration that the link of the ithtechnology and the jth technology is denoted as IRLij,l3 ,and the belief of reaching IRLij,l3 is εij,l3 . The IRL

for the integration is {(IRLij,l3 , εij,l3)},where i, j =1, 2, · · · , n and l3 = 1, 2, . . . , L3.

Defination 4 There would L (L = L1 × L1 × L3)possible combinations resulting from the belief distri-bution of the input data. The lth combination in be-lief structure would be {(TRLi,l1, εi,l1), (TRLj,l2, εj,l2),(IRLij,l3 , εij,l3)}

Sauser’s SRL assessment approach is still applied tocalculate the ISRL of the integration. For the lth com-bination, TRLi, TRLj and IRLij are used to representTRLi,l1 , TRLj,l2 and IRLij,l3 .

Defination 4.1 TRL is defined in (5) as a vector,where TRLi and TRLj are normalized to be between 0and 1.

[TRL]2×1 = [ TRLi TRLj ]Tnorm. (5)

Defination 4.2 IRL is defined in (6) as a matrix, whereIRLij is the IRL between the ith technology and the jthtechnology and the value of IRLij is normalized to be be-tween 0 and 1.

[IRL]2×2 =[

IRLii IRLij

IRLji IRLjj

]2×2

. (6)

Note that the value of IRLii/IRLjj denotes the hypo-thetical integration of the ith/jth technology to itself andIRLii/IRLjj = 1.

Defination 4.3 Calculate the SRL of the integrationwith each technology respectively, denoted as “ISRLi”and “ISRLj”. Equation (7) shows the calculation ofISRLi and ISRLj:[

ISRLi

ISRLj

]=[

IRLii IRLij

IRLji IRLjj

]2×2

×[

TRLi

TRLj

]=

[(TRLi + IRLijTRLj)/2(TRLj + IRLjiTRLi)/2

]. (7)

Note that the value of ISRLi/ISRLj would fall withinthe interval (0, 2) because the hypothetical integration ofthe ith/jth technology to itself is calculated. Thus thevalue of ISRLi/ISRLj must be divided by 2.

Defination 4.4 Calculate the component SRL of theintegration, denoted as “ISRL”, which is the average of“ISRLi” and “ISRLj”. Equation (8) shows the calcula-tion of ISRL:

ISRL =ISRLi + ISRLj

2=

(TRLi + TRLj)(1 + IRLij)4

. (8)

Defination 4.5 Determine the level of ISRL. Com-pare the result of (8), ISRL, with the definitions in Table


1 of Section 2.2.1. The derived ISRL is then presentedas {ISRLk,l, βkl}, where l denotes the lth combinationof TRLi, TRLj and IRLij , k denotes the kth level thatachieving by ISRL, l = 1, 2, . . . , L, k = 1, 2, . . . , 5 andL = L1, L2, L3.

3.3 ER algorithm for synthesis

The next step is to synthesize all of the ISRLs of the L

combinations into a composite ISRL of the integration us-ing the ER algorithm.

The ER algorithm is based on a multi-attributes assess-ment framework and D-S evidence synthesis theory. TheER algorithm was proposed by [11,12], which has beenwidely applied in the decision making, safety and risk as-sessment, organizational self-evaluation, and supply chainproblems in the engineering design. The ER algorithm ap-pears as though it has instinctive insights when address-ing the inadequate information, which enriches the finalassessment.

Three steps need to be performed to synthesize L com-binations of the input data to obtain the ISRL for the in-tegration. Step 1 is the preparation for the synthesis whileStep 2 and Step 3 are the process of the ER algorithm.

Step 1 For the lth combination of input data, normal-ize the probability of the lth ISRL denoted as wl, and theadjusted βkl denoted as βkl.

The normalized probability of the lth ISRL is calcu-lated by

wl = αl/

L∑l=1

αl (9)

where αl is the probability of obtaining the ISRL for thekth combination.

The adjustment of the belief distribution of ISRL canbe calculated by

βkl =

⎛⎝βkl

∑t=i,j,ij

|At|∑l=1

εt,l

⎞⎠ /3 (10)

where |At| is the number of the possible values ofTRL/IRL for the ith/jth technology/the ijth integration,and εt,l is the belief of obtaining the value TRL/IRL as At

for the ith/jth technology /the ijth integration.

If the input data are adequate, then|At|∑l=1

εt,l =

1, ∀t, βkl = βkl.Step 2 Construct the basic probability mass and ob-

tain the matching degree of the assessment goal on all ofthe combinations of the input data.

Transform the belief degrees for all of the values intobasic probability masses using the ER algorithm, the coreof which is to derive the matching degree of the assessment

goal on all of the combinations by clustering the matchingdegrees of the input data on the combinations. Equations11−14 shows the calculation process:

mk,l = ωlβkl (11)

mR,l =

(1 − ωl

5∑k=1

βlk

)(12)

mR,l = 1 − ωl (13)

m̃R,l = ωl

(1 −

L∑l=1

βkl

)(14)

where mR,l = mR,l + m̃R,l, for all l = 1, . . . , L.

The probability for the assessment result for the kthcombination consists of two parts. mR,l is caused by therelative importance of the input data. m̃

R,lis caused by the

inadequacy of the input data.Step 3 Synthesize the result of L combinations and

obtain the belief of achieving the kth level of the result,denoted as βk. Gather the inadequate information togetherand store it in one variable, denoted as βH .

Use mR,l, mR,l and m̃R,l to represent the united beliefdistribution for the result from the input information. Weapply the analytical form of the ER algorithm [14] for theinput information synthesis, as shown in (15)−(20). Theiterative form can be found in [11,12,14].

mk =N

[L∏

l=1

(mk,l +mR,l + m̃R,l)−L∏

l=1

(mR,l +m̃R,l)

](15)

m̃H = N

[L∏

l=1

(mR,l + m̃R,l) −L∏

l=1

(mR,l)

](16)

mH = N

[L∏

l=1

(mR,l)

](17)

N =

[5∑

k=1

L∏l=1

(mk,l + mR,l + m̃R,l)−

(5 − 1)L∏

l=1

(mR,l + m̃R,l)]

(18)

βk = ml/(1 − mH) (19)

βH = m̃H/(1 − mH) (20)

where βk is the belief of achieving the kth level of all ofthe five possible levels and βH is the belief of the inade-quacy information.

Note again that the ER algorithm is used twice in thisstudy, first to synthesize all of the ISRLs of L combina-tions into one composite ISRL, as demonstrated in thissection. Then to synthesize all of the ISRLs into the sys-tem’s SRL, which would be the same in terms of the cal-culation and need not be repeated.


4. Case study

4.1 Case study 1

The following is a case that was studied by [9] and thesame data are used for comparison. A system is composedof three technologies, denoted as Tech1, Tech2 and Tech3and there is an integration between Tech1 and Tech2, de-noted as IRL1,2, and the integration between Tech2 andTech3, denoted as IRL2,3 as shown in Fig. 1.

Fig. 1 A system with three technologies and two integrations

Four conditions of Case 1 are studied. The data of Case1A are adequate, which are the same as those of [9]. Thereis 10% inadequacy in Case 1B, obtained by multiplyingthe data of Case 1A with 0.9. There is 20% inadequacyin Case 1C, obtained by multiplying the data of Case 1Awith 0.8. There is 50% inadequacy in Case 1D, obtainedby multiplying the data of Case 1A with 0.5. All of thedata are listed in Table 2.

The calculation steps of ISRL1,2 in Case 1B are shownin Tables 3−6 as an example. For Tables 3−4, only thefirst 5 rows of all L (L = 27) are listed. The result of Case1B is demonstrated by using IDS [15], as shown in Fig. 2.

ISRL of the integration between Tech1 and Tech2 inthe belief structure is ISRL1,2 = {(1, 0.77%), (2, 92.49%),

Table 2 Input data for case study 1 in four conditions

Condition A Condition B Condition C Condition D

Technology Belief structure Belief structure Belief structure Belief structure

1 {(7,0.1),(8,0.8),(9,0.1)} {(7,0.09),(8,0.72),(9,0.09)} {(7,0.08),(8,0.64),(9,0.8)} {(7,0.05),(8,0.4),(9,0.05)}2 {(5,0.2),(6,0.7),(7,0.1)} {(5,0.18),(6,0.63),(7,0.09)} {(5,0.16),(6,0.56),(7,0.08)} {(5,0.1),(6,0.35),(7,0.05)}3 {(6,0.1),(7,0.6),(8,0.3)} {(6,0.09),(7,0.54),(8,0.27)} {(6,0.08),(7,0.18),(8,0.24)} {(6,0.05),(7,0.3),(8,0.15)}

Integration Belief structure Belief structure Belief structure Belief structure

1,2 {(1,0.8),(2,0.15),(3,0.05)} {(1,0.72),(2,0.135),(3,0.045)} {(1,0.64),(2,0.12),(3,0.04)} {(1,0.4),(2,0.075),(3,0.025)}2,3 {(6,0.3),(7,0.6),(8,0.1)} {(6,0.27),(7,0.54),(8,0.09)} {(6,0.24),(7,0.48),(8,0.08)} {(6,0.15),(7,0.3),(8,0.05)}

Table 3 βkl, αl and wl

l ISRLl β1l β2l β3l β4l β5l αl wl

1 0.370 4 1 0 0 0 0 0.011 7 0.016

2 0.401 2 0 1 0 0 0 0.040 8 0.056

3 0.432 1 0 1 0 0 0 0.005 8 0.008

4 0.407 4 0 1 0 0 0 0.002 2 0.003

5 0.441 4 0 1 0 0 0 0.007 7 0.010 5

Table 4 mk,l , mR,l, mR,l and m̃R,l

l m1,l m2,l mR,l mR,l m̃R,l

1 0.014 4 0 0.985 6 0.984 0.001 6

2 0 0.050 4 0.949 6 0.944 0.005 6

3 0 0.007 2 0.992 8 0.992 0.000 8

4 0 0.002 7 0.997 3 0.997 0.000 3

5 0 0.009 45 0.990 55 0.989 5 0.001 05

Table 5 mk, mH and mH

m1 m2 m3 m4 m5 m̃H mH

Value 0.005 3 0.633 9 0 0 0 0.314 6 0.046 2

Table 6 βk and βH

β1 β2 β3 β4 β5 βH

Value/(%) 0.77 92.49 0 0 0 6.74

(N/A, 6.74%)}. Following the same calculation steps,the ISRL of the integration between Tech2 and Tech3in the belief structure is obtained as ISRL2,3 ={(2, 13.03%), (3, 79.70%),(N/A,7.27%)}.

Fig. 2 Assessment results for Case 1B

Fig. 3 Assessment results for Case 1 under four conditions


A composite demonstration of Case 1 under four condi-tions is presented as shown in Fig. 3, in which the “beliefdegree” is denoted by βk and the “unknown” in SRL isdenoted by N/A.

A primary conclusion is drawn through the comparisonamong the results of Case 1 under four conditions:

(i) The effectiveness of the proposed approach is va-lidated. Where there is an inadequate information in theinput (Case 1B/C/D), there is an inadequate information inthe assessment results. This is direct proof of the efficiencyof the proposed approach.

(ii) The consistency among SRLs of Case 1 under thefour conditions is validated. As shown in Fig. 4, level “2”stays as the most likely result regardless of the probabilityof the inadequacy in different conditions. Furthermore, theresults of Case 1 under four conditions have the same com-parative proportion regarding the belief distribution amongdiffe-rent assessment grades. This is because the input datafor Case 1 under the four conditions are artificially gener-ated and therefore have the same comparative proportionof the belief distribution.

(iii) The nonlinear correlation between the growth of theadequacy/inadequacy in the input data and the growth ofthe adequacy/inadequacy in the assessment result is vali-dated. This is caused by the nonlinear nature of the ERalgorithm [17].

(iv) The robustness of the proposed approach is vali-dated. The nonlinear feather of the proposed approach alsoleads us to study the robustness. With the inadequacy in theinput growing from “0%” to “10%”, “20%”, and “50%”,the inadequacy in the assessment result grows from “0%”

to “6.43%”, “13.08%”, and “35.84%”, respectively. Thegrowth of the inadequate information in the assessment re-sult, which is the primary concern in this study, is muchslower than that in the input.

4.2 Case study 2

4.2.1 Condition A

Case 2 has also been studied by [4, 7−9]. Confronted witha series of problems, a robot servicing mission (RSM) hadbeen commenced by NASA to repair the Hubble spacetelescope. A concept diagram of the RSM is shown inFig. 4. Information on each technology can be found in[4].

Fig. 4 System concept diagram

The input is the same as that by [9], with the data multi-plied by 0.9 to create 10% inadequate information, as listedin Table 7.

The ISRL of each integration is listed as in Table 8.And SRL of the system in the belief structure for Case2A is {(1, 4.64%), (2, 25.01%), (3, 64.45%), (4, 0.03%),(N/A, 5.87%)}. A graphic presentation of SRLs for Case2A can also be found in the belows.

Table 7 Input data in belief structure for Case 2A

Technology Belief structure Integration Belief structure

1 {(7,0.135),(8,0.72),(9,0.045)} 1,2 {(4,0.045),(5,0.54),(6,0.315)}2 {(7,0. 045),(8,0.72),(9,0.135)} 1,3 {(5,0.18),(6,0.63),(7,0.09)}3 {(6,0.09),(7,0.72),(8,0.09)} 2,3 {(5,0.045),(6,0.72),(7,0.135)}4 {(5,0.135),(6,0.72),(7,0.045)} 2,4 {(4,0.09),(5,0.72),(6,0.09)}5 {(5,0.225),(6,0.63),(7,0.045)} 3,5 {(5,0.18),(6,0.63),(7,0.09)}6 {(5,0.18),(6,0.63),(7,0.09)} 4,5 {(1,0.135),(2,0.54),(3,0.225)}

5,6 {(2,0.36),(3,0.54)}

Table 8 ISRL of each integration for Case 2A

ISRL Result in belief structure

ISRL12 {(2, 0.02%), (3, 93.31%), (4, 0.13%), (N/A, 6.55%)}ISRL13 {(2, 0.14%), (3, 93.16%), (4, 0.02%), (N/A, 6.67%)}ISRL23 {(2, 0.01%), (3, 93.03%), (4, 0.10%), (N/A, 6.86%)}ISRL24 {(2, 12.85%), (3, 79.08%), (N/A,8.06%)}ISRL35 {(2, 32.89%), (3, 58.78%), (N/A, 8.34%)}ISRL45 {(2, 27.48%), (3, 64.48%), (N/A, 8.03%)}ISRL56 {(1, 11.45%), (2, 81.26%), (N/A, 7.29%)}

4.2.2 Condition B

Case 2B is studied by concerning the structure of the sys-tem with the same scenarios as in Condition A becausethe structure of a system also plays a vital role in a sys-tem along with components that physically comprise thesystem [21]. Therefore, it is necessary to take the struc-ture of the system into consideration when performing anysystem-level assessment.

As shown in Fig. 5, the system consists of two parts,named as “Part 1” and “Part 2”. The SRLs for Part 1, Part


2 and the system are calculated and given as follows:SRLpart1 = {(2, 2.21%), (3, 93.13%), (4, 0.04%),

(N/A, 4.62%)}SRLpart2 = {(1, 11.46%), (2, 65.46%), (3, 16.62%),

(N/A, 6.45%)}SRL = {(1, 5.27)(2, 31.74%), (3, 57.97%), (4, 0.02%),

(N/A, 4.99%)}

Fig. 5 System concept diagram with the structure

The results are also demonstrated by using IDS [15], asshown in Fig. 6.

Some preliminary conclusions are drawn as follows:(i) The effectiveness of the proposed approach is vali-

dated and the consistency is further verified. When thereis an inadequate information in the input, there is a corre-spondingly inadequacy in the assessment results. The be-lief distribution stays the same. This is because the inputdata are the same. The consistency of results proves theeffectiveness of the proposed approach.

Fig. 6 Assessment results for Case 2A/2B

(ii) There are changes in the belief degrees for all evalu-ation grades. The belief degree for grade 2 increases from“25.01%” in Case 2A to “31.74%” in Case 2B. The answercan be found if we note that the belief degree for grade 2 ofSRLpart2 is “65.46%”, which makes a huge impact on thefinal result. It can also be found that the belief degree forgrade 3 decreases from “64.45%” of Case 2A to “57.97%”of Case 2B.

(iii) The inadequacy in assessment results decreaseswhen the structure of the system is taken into considera-tion. The “N/A” information in the assessment results de-creases from “5.87%” in Case 2A to “4.99%” in Case 2B.It can be explained that the ambiguity in the assessment isreduced when there is more information on the system.

To summarize, when considering the structure of thesystem, the impact of certain specific characteristics in theindividual technology/integration will be amplified, andthis change will reflect on assessment results. Therefore,with a better understanding of the system’s structure, theambiguity that is caused by the inadequacy of the input in-formation would be lessoned. However, because the inputdata are the same, the nature of the question stays the same,thus the comparative proportion of the belief distributionfor each grade in the assessment result stays the same.

4.3 Comparison with classical SRL

We would also like to make a comparison between the pro-posed approach and classical SRL approaches.

(i) Different types of the information

The biggest difference between the proposed approachand classical SRL (Sauser [3] and Tan [8,9]) is that theycan handle different types of information, as shown inFig. 7.

Fig. 7 Difference among approaches

Sauser’s SRL assessment is founded on the assumptionthat TRL, IRL and SRL are unique determined values,meaning that the input information is certain and adequate.Tan’s SRL assessment is founded upon the assumption thatthe values of TRL, IRL and SRL follow a probabilistic dis-tribution and thus take the uncertain and adequate infor-mation into consideration. The proposed approach in this


study mainly deals with the uncertain information includ-ing inadequacy, which differentiates this approach fromthose of Sauser’s and Tan’s essentially.

(ii) The consistency in simple cases

First the result of Case 1 with that of [9] since their dataare the same. Tan obtained the result with a mean valueof SRL as 0.55 and α =10% confidence interval as [0.50,0.60]. By comparing the assessment with standards thatare provided in Table 2, we can translate the result intothe following: the mean value of the composite SRL islevel 2 and the confidence interval α =10% is also level 2.This result is consistent with that of the proposed approach,which has 59.10% (highest probability) of certainty atlevel 2.

(iii) The conservation of classical SRL approaches incomplex cases

Case 2 describes a more complex scenario. The resultsof Case 2A/B both point to the level 3 while the resultof [8,9] directs to the level 2, whose results are the meanvalue of SRL as 0.548 and α = 10% confidence interval as[0.453, 0.662]. This means that classical SRL approachestend to be more conservative when dealing with complexscenarios.

5. Conclusion

The study is motivated by the inability to handle the in-adequate information in current SRL assessment. Anintegration-centric ER-based approach is proposed to meetthis challenge. The integration-centric perspective pro-vides a different angle for the SRL assessment, and alsoreduces the computational complexity. The ER algorithmis applied to extract the inadequate information from theinput data and store them in the assessment result, denotedas N/A. Two cases are studied to validate the effectivenessof the proposed approach: if there is an inadequate infor-mation in the input data, then the inadequate informationwill appear in assessment results. The nonlinear correla-tion between assessment results and the input data has beenrevealed as well.

Certain interesting features are detected by the “contra-dictory results” from the two case studies. The assess-ment results of Case 1 under four conditions are consistent,when the input data are different and the structure of thesystem is not concerned. The assessment results of Case 2under two conditions are inconsistent, when the input dataare exactly the same and the structure of the system is con-cerned.

This feature shows that the structure of the system,along with the input data, plays an important role in af-fecting the assessment results. Special attention should

be paid to the change of the inadequate information, orN/A: the inadequacy decreases from “5.87%” in Case 2Ato “4.99%” in Case 2B when the structure of the systemis concerned. It can be explained that the structure of thesystem can be recognized as a type of “information”, sowhen the structure of a system is concerned, the input forthe SRL assessment is strengthened and the inadequacy inSRL assessment result is decreased. Since this is only anobservation-based deduction, more practical and theoreti-cal work is required to prove its rationality and validity.

References

[1] J. C. Mankins. Technology readiness assessment: a new ap-praoch. Acta Astronautica, 2009, 65(3): 1208–1215.

[2] J. C. Mankins. Technology readiness assessment: a retrospec-tive. Acta Astronautica, 2009, 65(3): 1216–1223.

[3] B. Sauser, J. Ramirez-Marquez, D. Verma. From TRL to SRL:the concept of systems readiness levels. Proc. of the 4th Con-ference on Systems Engineering Research, 2006: 652–662.

[4] B. Sauser, J. Ramirez-Marquez, D. Henry, et al. A system ma-turity index for the systems engineering life cycle. Interna-tional Journal of Industrial and Systems Engineering, 2008,3(6): 673–691.

[5] B. Sauser, J. Ramirez-Marquez, R. Magnaye, et al. A systemsapproach to expanding the technology readiness level withindefense acquisition. International Journal of Defense Acquisi-tion Management, 2008, 1(3): 39–58.

[6] R. Gove, B. Sauser, J. Ramirez-Marquez. Integration maturitymetrics: development of an integration readiness level. Infor-mation Knowledge System Management, 2010, 9(1): 17–46.

[7] R. Gove. Development of an integration ontology for systemoperational effectiveness. Hoboken, USA: Stevens Institute ofTechnology, 2007.

[8] W. Tan, B. Sauser, J. E. Ramirez-Marquez. Monte-Carlo simu-lation approach for system readiness level estimation. Proc. ofthe 19th International Symposium of the International Councilon Systems Engineering (INCOSE), 2009: 324–336.

[9] W. Tan, J. E. Ramirez-Marquez, B. Sauser. A probabilistic ap-proach to system maturity assessment. Journal of System En-gineering, 2011, 14(3): 279–293. (in Chinese)

[10] R. Phaal, P. J. Palmer. Technology management-structuring thestrategic dialogue. Engineering Management Journal, 2010,22(1): 64–74.

[11] J. B. Yang, J. Liu, J. Wang, et al. Belief rule-based inferencemethodology using the evidential reasoning approach. IEEETrans. on Systems, Man, and Cybernetics-Part A: Systems andHumans, 2006, 36 (2): 266–285.

[12] J. B. Yang. Rule and utility based evidential reasoning ap-proach for multiple attribute decision analysis under uncer-tainty. European Journal of Operational Research, 2001,131 (1): 31–61.

[13] G. A. Shafer. A mathematical theory of evidence. New Jersey:Princeton University Press, 1976.

[14] Y. M. Wang, J. B. Yang, D. L. Xu. Environmental impact as-sessment using the evidential reasoning approach. EuropeanJournal of Operational Research, 2006, 174(3): 1885–1913.

[15] D. L. Xu, G. McCarthy, J. B. Yang. Intelligent decision sys-tem and its application in business innovation self assessment.Decision Support System, 2006, 42(27): 664–673.

[16] J. Jiang, X. Li, Z. Zhou, et al. Weapon system capability as-sessment under uncertainty based on the evidential reasoningapproach. Expert Systems with Applications, 2011, 38(11):13773–13784.


[17] J. B. Yang, D. L. Xu. Nonlinear information aggregation viaevidential reasoning in multiattribute decision analysis underuncertainty. IEEE Trans. on Systems, Man, and Cybernetics,2002, 32(3): 376–393.

[18] S. R. Sadin, F. P. Povinelli, R. Roser. NASA technologypush towards future space mission systems. Acta Astronautica,1989, 20(2): 73–77.

[19] J. Smith. An alternative to technology readiness level for non-developmental item (NDI) software. Pittsburgh: Carnegie Mel-lon Press, 2004.

[20] F. Redmill. Exploring subjectivity in Hazard analysis. Engi-neering Management Journal, 2002, 12(3): 139–144.

[21] Z. G. Tao, M. Z. Wang. Study on system of systems capabilityarchitecture development based on DoDAF V2.0. InternationalJournal of System of Systems Engineering, 2012, 3(1): 47–59.

Biographies

Leilei Chang was born in 1985. He receivedhis M.S. degree from National University of De-fense Technology, China. He is currently a doc-toral candidate in National University of De-fense Technology. His research interests includeweapon system of systems, technology systemof systems and RIMER.E-mail: [email protected]

Mengjun Li was born in 1964. He receivedhis M.S., Ph.D. degrees from National Uni-versity of Defense Technology, China. He iscurrently a professor in National Universityof Defense Technology. His research inter-ests include problems in management and de-cision theories in weapon acquisition.E-mail: [email protected]

Ben Cheng was born in 1982. He received hisM.S. degree from National University of De-fense Technology, China. He is currently a doc-toral candidate in National University of De-fense Technology. His research interests includecapability evaluation problems in weapon sys-tem of systems.E-mail: [email protected]

Ping Zeng was born in 1987. He received his M.S. degree from National University of DefenseTechnology. He is currently a doctoral candidatein National University of Defense Technology,China. His research interests include weaponplanning in weapon system of systems.E-mail: [email protected]

integration-centric approach to system readiness assessment based on evidential reasoning

Documents