CSER 2015 March 18-19, 2015 1

A Research on Measuring and Reducing Problem Complexity to Increase System Affordability

From Theory to PracticeBy

Alejandro Salado and Roshanak Nilchiani13th Annual Conference on Systems Engineering Research (CSER)

March 18, 2015

Stevens Institute of TechnologyHoboken, NJ

www.stevens.edu/sse/CSER2015org

CSER 2015 March 18-19, 2015 2

System affordbility

AFFORDABILITYPERCEIVED VALUE

BUDGETPerceived benefit Actual benefit Cost

Business Marketing Technology push

Process improvementLabor

OutsourcingProcurement strategies

Investment

System development &

operation

DfEDfA

Psychology-based design

Spiral

Context/Environment

AwarenessLobby Market change

Economic situationLaws and regulations

AwarenessLobby

Competition

Desig

n&De

velo

p.

Trad

espa

ce d

efini

tion Use cases

Pugh matricesSpiral, agile, leanOther tradit. SE

De-scope needsUse casesAgile, lean

Pugh matricesOther tradit. SE

NbCTradespace maximization/adaptation

CertificationQuality

Process improvement6-sigma

Servicing Customer service

Evidence-based deliverableSkunkworks

Evidence-based deliverableSkunkworks

DfR, DfA, DfE,DfS, DfF, Value-driven design

DtC, DfC, TCO, Costreviews, DfM, Complexity

control

Pareto SetTradespace exploration

Man

uf.

Ops

.PM

Manuf.: Manufacturing - Ops.: Operation - PM: Project Management

CSER 2015 March 18-19, 2015 3

Research design: An end-to-end approach

CSER 2015 March 18-19, 2015 4

Research design: An end-to-end approach

CSER 2015 March 18-19, 2015 5

Foundations: Theorems

A. Salado, R. Nilchiani, and D. Verma, “A formal theory of requirements engineering: stakeholder needs, system requirements, solution spaces, and requirements’ qualities,” unpublished.A. Salado and R. Nilchiani, “A mathematical justification for increasing the size of the solution space to improve the probabilities of designing compliant and affordable systems,” unpublished.

VARIABLES RELATION

System requirements

#Solution space

#Solution space

#Solution space

#Solution space

Conflicting requirements

Difficulty of compliance

Difficulty of affordability

Monotonic

Monotonic

Monotonic

Monotonic

CSER 2015 March 18-19, 2015 6

Excess requirements need to be compensated with effort...

A. Salado and R. Nilchiani, “Increasing the probability of developing affordable systems by maximizing and adapting the solution space,” Procedia Computer Science, Vol. 28, 2014, pp. 547-554.

CSER 2015 March 18-19, 2015 7

Consequence of foundations: Problem complexity

A. Salado and R. Nilchiani, “The Concept of Problem Complexity,” Procedia Computer Science, Vol. 28, 2014, pp. 537-546.

CSER 2015 March 18-19, 2015 8

From foundations to practice

Traditionally…

Requirement categories employed to facilitate completeness of a req set

Written from a designer perspective or procurement perspective

Sets of more tan 1,000 requirements, where each requirement COSTS

1. Because of their individual management

2. Because of their effect in unnecessarily reducing the solution space

…but feel for completeness (not completeness) is paid at a high price

Mix product and process requirements

Facilitate redundant and overlapping requirements

Facilitate elicitation of solution constraints, not problem definition

Facilitate elicitation of requirements to enabling systems, not to the system

CSER 2015 March 18-19, 2015 9

An excerpt from an actual space system 35 requirements!

A. Salado and R. Nilchiani,“A categorization model of requirements based on Max-Neef‘s model of human needs,“ Syst. Eng., 17:348-360,2014

CSER 2015 March 18-19, 2015 10

A method for practitionersValue level

Functions (Do)

Performance (Being)

Resources (Have)

Interaction (Interact)

Break-event

Req. 1Req. 2Req. 3

Req. 4 Req. 5Req. 6

Req. 7

Goal Req. 8 Req. 9 Wish Req. 10 Req. 11

Req. 12

Functional requirements (Do)What the system does in essence, which includes what it accepts and what it delivers Performance requirements (Being): How well the system does it, which includes performance related to functions the system performs or characteristics of the system on its own, such as –ilitiesResource requirements (Have): What the system uses to transform what it accepts in what it deliversInteraction requirements (Interact): Where the system does it, which includes any type of operation during its life-cycle.

A partition!No overlap of categories

Full system definition

Requirements as subsetsAvoids flaw of individual

prioritizationReflects value sets to

stakeholders

Bigger picture understsanding

Facilitates relating functions to their performance,

resources, and environment

Resources is moreFacilitates allocation of

unnecessary constraints for easy identification

A. Salado and R. Nilchiani,“A categorization model of requirements based on Max-Neef‘s model of human needs,“ Syst. Eng., 17:348-360,2014

CSER 2015 March 18-19, 2015 11

Measuring its effectiveness: an experiment with practitioners

INCLUSIONPracticing SE

Non-practicing SE >5y SE expResearcher SE

EXCLUSIONStudents

Not large-scale systemsNon SE

Non-parametric Mann-Whitney U Test95% confidence / alfa = 0.05

ORGANIZATION13 test + 8 control + 3 discarded

Random assignmentIsolation

NON-MANIPULATED INDEP.Experience/Competence

Specific knowledge

DEPENDENTAmount of constraints

Amount of inapplicable reqsCompleteness

INDEPENDENTCategorization method

A. Salado and R. Nilchiani,“Reducing excess requirements through orthogonal categorizations: results of a factorial experiment ,“ unpublsihed, 2014.

CSER 2015 March 18-19, 2015 12

Measuring its effectiveness: an experiment with practitioners

MOE Problem statement

Self-perception on req quality

Experience in systems

engineering

1.H0 0.220 0.038 -0.055

2.H0 0.459* -0.166 -0.129

3.H0 -0.408 0.579 0.570

NULL HYPOTHESES

SECOND H0Both groups elicit the same amount

of inapplicable reqs.

THIRD H0Both groups elicit the same amount

of net requirements.

FIRST H0Both groups elicit the same

amount of constraints.

p = 0.001 0.025 0.804 Statistical power

> 97%

median = 26% vs 5% 18% vs 1% 24 vs 25 26% vs 0%

CompletenessSame level of completenessUnnecessary constraints

Less unnecessary constraintsInapplicable requirements

Less inapplicable requirements

!

A. Salado and R. Nilchiani,“Reducing excess requirements through orthogonal categorizations: results of a factorial experiment ,“ unpublsihed, 2014.

CSER 2015 March 18-19, 2015 13

Identifying conflicting requirements

Expert assessment ? MBSE

- Low effectiveness- Low effort- Before architecture

- High effectiveness- High effort- Design exists

CSER 2015 March 18-19, 2015 14

Tension matrix and elemental decomposition

Reqs. Resources Phases of matter

Elemental decomposition

Laws of

physics Laws of society

Logical

r7 r8 r9 S L G V T P v L1 L2 L3

F

r1 X ↑ Methods

in

chapter 2

r2 X ↑ ↓

r3 X

P

r4 ↓

r5 X ↑ ↓ ↑

r6 X ↑

R

r7

r8 ↓

r9

I

r10 X

r11 X ↓ ↓

r12 X

HeuristicsTargeted modeling

Elemental decomposition

CSER 2015 March 18-19, 2015 15

Identifying conflicting requirements

NEEDS- IR band imaging- X band imaging- PtP secure coms

r31 r32 S L G V Pc Temp OpTime Rfout Rfnoise R0 Data

The satellite shall perform X-bandoperations at 30° incidence angle.The satellite shall operate for more than 5 X X L IThe satellite shall have a reliability higher L LThe satellite shall transmit communicationservices at 7.75 GHz.

I

The satellite shall transmit image data at IThe satellite shall receive communicationservices at 400 MHz.

L

The satellite shall transmit image datawith EIRP higher than 30 dBW.

I I

The satellite shall transmit communicationservices with EIRP higher than 35 dBW.

I I

The satellite shall have a G/T higher than -9 dB/K for communication services.

L

The satellite shall transmit telemetry data IThe satellite shall receive command data LThe satellite shall transmit telemetry datawith EIRP higher than 13 dBW.

I I

The satellite shall have a G/T forreceiving command data higher than -49

L

The satellite shall image the Earth withspatial resolution better than 30 m.Note: applicable to IR band.

X X L I

The satellite shall image the Earth with afield of view higher than 1.22°.Note: applicable to IR band.

I

The satellite shall image the Earth withSNR higher than 125Note: applicable to IR band.

L

The satellite shall have a radiometricaccuracy better than 2 unit.Note: applicable to IR band.The satellite shall image the Earth withspatial resolution better than 4 m.Note: applicable to X band in range andazimuth directions.

I

The satellite shall image the Earth with aswath of no less than 40 km.

X I

The satellite shall have a lower than -18 I IThe satellite shall store image data for upto 4 h.

X X I

Req ID

Requirement

Elementary decompositionLaws of physics

Resources Phases of matter

Perf

orm

ance

Cate

gory

0 10 20 30 40 50 600

0.51

1.5

IR spatial resolution (m)

Utilit

y

COMPARATIVE ANALYSISBenchmark vs Tension matrix

1.5 1.55 1.6 1.65 1.7 1.75 1.8

x 106

0.5

0.52

0.54

0.56

0.58

0.6

0.62

0.64

0.66

0.68

Cost (k$)

Util

ity

CSER 2015 March 18-19, 2015 16

Wrapping up!

Continuous satisfaction levels? Absolute

calibration?Empirical metrics for

improvement?