pendar webinar 1 with notes

© Predictable Network Solutions Ltd 2017 www.pnsol.com

PEnDAR Focus Group

Performance Ensurance by Design, Assuring RequirementsFirst Webinar


Click to edit Master title style 2

Goals• Consider how to enable Validation & Verification of

cost and performance • For distributed and hierarchical systems • via sophisticated but easy-to-use tools• Supporting both initial and ongoing incremental

development

• Provide early visibility of cost/performance hazards• Avoiding costly failures• Maximising the chances of successful in-budget delivery

of acceptable end-user outcomes

Partners

Supported by:

PEnDAR project



Our offer to youLearn what we know about the laws of the distributed world:• Where the realm of the possible lies;

• How to push the envelope in a way that benefits you most;

• How to innovate without getting your fingers burnt.

Get an insight into the future:• Understand more about achieving

sustainability and viability;• See where competition may lie in the future.

Our ‘ask’Help us to find out:• In which markets/sectors/application

domains will the benefits outweigh the costs?• Which are the major benefits?

• Which are just ‘nice to have’?

• How could this methodology be consumed?• Tools?• Services?

Role of the focus group


The system delivery challenge

© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com

5

System delivery

challengeSystem

Requirements

Tim

e pr

essu

reCom

plexity

Cost/resource constraints

Forces hierarchical decomposition

Forces parallel development

Change during development

Subcontracted subsystems

Use of COTs componentsVague/contradictory

Leave ‘hard’ problems to the end

Existing asset re-use constraints



How to specify and verify subsystem

requirements/acceptance criteria

Functional outcomes

Performance outcomes

Resource constraints

MEE

T

Risks

MANAGEPeak Loads

Integration testing

Rework

MIN

IMISE

Post-deployment scaling

System delivery challenge


7

Constrained performance hazards

Unconstrained performance

hazards

Distributed

Centralised

Dedicated SharedResources

Stru

ctur

e

Traditional avionics

Cloud apps

Traditional mainframe

Standalone microcontrollers

Modern avionics

Client-server systems

Established approaches running out of steam?


Dimensions of the challenge


9

Out

com

es

Deliv

ered

Resources Consum

ed

Individual impact

Variability

Exception/failure

Syste

m im

pactSelf-induced

Externally created

Indi

vidua

l impa

ct Mitigation

Correlation

System impact

Propagation

Scale

Schedulability

CapacityDistance

Number

Tim

e

Space Density


10

Out

com

es

Deliv

ered

Resources Consum

edVariability

Exception/failure

Scale


11

Out

com

es

Deliv

ered

Resources Consum

edVariability

Exception/failure

Scale

Distance

Number

Tim

e

SpaceSchedulability

Capacity

Density


12

Out

com

es

Deliv

ered

Resources Consum

edVariability

Exception/failure

Mitigation

Propagation

Scale

Schedulability

CapacityDistance

Number

Tim

e

Space Density


13

Out

com

es

Deliv

ered

Resources Consum

edVariability

Exception/failure

Self-induced

Externally created

Mitigation

Correlation

Propagation

Scale

Schedulability

CapacityDistance

Number

Tim

e

Space Density


14

Out

com

es

Deliv

ered

Resources Consum

edVariability

Exception/failure

Self-induced

Externally created

Individual impactIn

divid

ual im

pact Mitigation

Correlation

Syste

m im

pact

System impact

Propagation

Scale

Schedulability

CapacityDistance

Number

Tim

e

Space Density


15

Out

com

es

Deliv

ered

Resources Consum

ed

Individual impact

Variability

Exception/failure

Syste

m im

pactSelf-induced

Externally created

Indi

vidua

l impa

ct Mitigation

Correlation

System impact

Propagation

Scale

Schedulability

CapacityDistance

Number

Tim

e

Space Density


Practical methodology



Support stages of the SDLC• Design

• Feasibility analysis• Hierarchical decomposition• Subsystem acceptance criteria

• Verification• Checking delivery of quantified outcomes• Evaluating resource usage• Re-verification during system lifetime

• Validation• Quantification of performance criteria• Checking coverage and consistency

Quantify hazards• Failure to meet outcome requirements

• Physical constraints• Schedulability constraints• Supply chain constraints

• Failure to meet resource constraints• Scaling• Correlations

Interaction with System Development Life Cycle

© Predictable Network Solutions Ltd 2017

18

www.pnsol.com

Quantifying intent

• The key challenge is to establish quantified intentions• For outcomes/resources/costs

• Then a variety of mathematical techniques can be applied• queuing theory• large deviation theory• ∆Q algebra

• This is “only rocket science”• Not brain surgery!


19

www.pnsol.com

POLL

Imagine two technical scenarios:• In the first, a client asks for “a replacement for its phone system”.• In the second, a client asks for “a VoIP system with an average MOS of

4.2, the ability to take 30 concurrent calls, a failover system for when it hits capacity (e.g. voicemail/recorded message), first priority for emergency calls and second priority for current calls”.

If the first scenario is 1 and the second is 5, where on a scale of 1 to 5 would you place the requirements you generally encounter in your field?


20

www.pnsol.com20

Quantifying timelinessOutcome requirement:Suppose we could get the marketing department to specify how long it’s acceptable to wait for the first frame of a video:• 50% of responses within 3 seconds• 95% of responses within 10 seconds• 99.9% of responses within 15s• 0.1% failure rate



• Suppose the black line shows the delivered CDF• From measurement, simulation or

analysis

• This is everywhere above and to the left of the requirement curve• This means that the timeliness

requirement is satisfied• If not, there is a performance

hazard

Meeting a timeliness requirement


Questionnaire analysisWho are you and what do you think?


23

012345

Management System Design / Architect

Capability development

/ Research

Marketing / Sales

0

1

2

3

4

0

2

4

6

8

What field do you work in?

What is your role/what level do you work at?

Which parts of the product lifecycle are you involved in?


24Where do system/product requirements come from?

1 2 3 4 50

1

2

3

4Precise and unambiguous?

1 2 3 4 50

1

2

3

4 Quantifiable and quantified?

0

2

4

6


25How / when does your organisation determine whether requirements have been / will be met?

0246

What is the typical response to the discovery that requirements will not be / have not been met?

0

1

2

3


26

www.pnsol.com

Importance of performance and resources

For the systems / products you produce / operate, how critical is performance to success?

4.4

To what extent is your role accountable for aspects of performance that are critical for success?

3.2

To what extent do you, in your role, have influence/control over how resources are shared?

2.8

Given that resources are shared, other parts of the system / other systems may consume resources on which you rely: to what extent is this a concern?

4.0

Given that resources are shared, your system / product may consume resources on which others rely: to what extent is this a concern?

3.8

My role is primarily concerned with the performance impact on the individual users / uses

My role is concerned with both the overall performance and the impact on individual users / uses

System performance is not a concern of my role


Summing up



Benefits• Avoid infeasible developments

• 'fail early’• prune blind alleys

• Address scalability early• including real-world constraints• avoid 'heavy tail’

• See the whole risk landscape• not just the 'first problem’

• Be able to write a safety case• Even for shared-resource distributed systems

Costs• Have to ‘quantify intent’

• may meet resistance

• Effort in adopting new tools and techniques• Upfront work needed before “real”

development starts• may not fit expectations/metrics of

‘progress’

Expected benefits and costs of this approach


Thank you!

pendar webinar 1 with notes

Technology