safety cases and arguments - a tutorial · pdf filesafety cases and arguments - a tutorial iet...

Safety Cases and Arguments - A TutorialIET Conference on System Safety and Cyber Security 2014

Matthew Squair

Jacobs Australia

14 October 2014

1 M.Squair (SSCS 2104) :

1 Introduction and overview

2 Methodology

3 What should be in a safety case?

4 Notations

5 Producing a safety case

6 Maintaining the safety case

7 Do safety cases work in the real world?

8 Limitations, advantages and disadvantages

9 Conclusions

2 M.Squair (SSCS 2104) :


2 Methodology


4 Notations





9 Conclusions

3 M.Squair (SSCS 2104) Introduction and overview:

What we’ll cover

Safety cases have become an integral, and perhaps little questioned, partof safety management...

Despite their prevalence there are concerns about both practicalapplication [Haddon-Cave 2009] and theoretical underpinnings

We’ll look at the theory and application of safety cases with a focus onarguments in the context of acquisition

We’ll also discuss the problems and limitations of safety cases


A very short history

Safety cases

Originated in the British chemical industry (CIMAH)

Applied to oil industry after Piper Alpha

Applied to UK Rail after Clapham junction

Have become part of the EU safety culture

Standards:I DEF-STAN 00-56I DEF (AUST) 5679I EUROCONTROL Safety case development manualI CMMI SAFE+I IEC 61508


Safety cases are a broad church

A Safety Case can be:

operational or design focused

part of a ’permit to operate’ regime

acquisition/project developed

goal (more usual) or rule/standard based*

*Safety cases have traditionally formed part of goal (performance) basedsafety regimes


There are other approaches

A MIL-STD-882 ’classic’ system safety program

Acquisition focused (customer-supplier)

Addresses proximal (system) causes of accidents

Safety Assessment Report is analogue ’ish’ to a safety case

Aerospace - certification focused, use of regulation and standards. Role ofevidence as the basis of certification

Nuclear - Use of Probabilistic Risk Assessment (PRA)


Why do it?

Various reasons

You may need a tool to manage operational safety

You may wish to reduce liability risk

The regulator may require as a ’permit to operate’

You may want to structure and organise safety documentation

You may want to communicate system risk to stakeholders

Be clear about the purpose

Different stakeholders may mean very different things when it comes tosafety cases, be clear about your purpose and who it serves when youprepare one


Some definitions

Safety argument. A safety argument is a clear, comprehensive anddefensible argument that explains how the available evidence supports theoverall claim of acceptable safety within a particular context [Kelly 1998]

Safety case. A safety case is a structured argument, supported by a bodyof evidence, that provides a compelling, comprehensible and valid casethat a system is acceptably safe for a given application in a givenenvironment (i.e a context) [MOD (UK) 2007]

Safety case report. The physical artifact(s) that presents the safetyargument and case. Normally the safety case report is not a standalonedocument and will refer out to supporting evidence.


Some definitions (cont’d)

Project safety case. Developed by a project to demonstrate the safety ofthe change that the project will introduce. Scope is normally the change,with a reliance on assumptions of pre-exsiting safety of the system intowhich the change is being introduced.

Unit safety case. Developed by a standing organisation (usually a serviceprovider) to demonstrate the safety of ongoing operations. Project safetycases can be subsumed into the unit safety cases of those systems theyaffect.



2 Methodology


4 Notations





9 Conclusions

11 M.Squair (SSCS 2104) Methodology:

Methodology [Bishop, Bloomfield 1998]

1 Identify safety requirements

2 Identify system architecture and outline the safety case

3 Assessment (preliminary) of concept design safety trades

4 Progressive elaboration of the design & safety case in parallel

5 Integrate into final safety case

6 Plan for long-term support infrastructure

7 Review and approval8 Long-term monitoring and audits

I of areas of concernI of support processesI to gather field evidence to support assumptions

9 Revise to reflect system and context changes


Where do safety cases fit?

Eurocontrol’s view



2 Methodology


4 Notations





9 Conclusions

14 M.Squair (SSCS 2104) What should be in a safety case?:

”Primarily the Safety Case is a matter of ensuring that every companyproduces a formal safety assessment to assure itself that its operations aresafe.

Only secondarily is it a matter of demonstrating this to a regulatory body.That said such a demonstration both meets a legitimate expectation ofthe workforce and the public and provides a sound basis for regulatorycontrol.”

Piper Alpha Inquiry Lord Cullen, 1990


Contents

Contains at a minimum[Kelly 1998]:

Supporting evidence on which the argument is based, becauseargument without evidence is unfounded

A high level argument, because evidence without argument isunexplained

I May include a number of separate sub-argumentsI A convergent conclusion as to the acceptability of the system

A meta-argument as to why the argument and evidence should bebelieved because both evidence and argument can be faulty[Hawkins et al., 2011]

Is the totality of the safety evidence NOT just a safety case report

Structure and organisation is essential to achieve clarity


Evidence

Information (facts or expert opinions) presented to show that the safetyargument is valid (ie true).

How much evidence is required and of what quality?

Some legal rules of evidence

Relevant (necessary to the argument, probative value)

Clarity (not confusing)

One interpretation (no potential for misleading)

Parsimonius (Not needlessly presenting cumulative evidence)

Authenticity (it is what it purports to be)

What is the standard of persuasion? (proof vs presumption, degrees ofcorroboration)

17 M.Squair (SSCS 2104) What should be in a safety case?: Evidence

Does this satisfy the legal rules of evidence?

Real evidence of a zonal hazard analysis


A Bayesian view of evidence

The probative value of evidence is intimately related to how ’close’ orintertwined it is with what you want to prove

Difference between direct (’smoking gun’) and backing (circumstantial)

Backing evidence is still important, use it to argue the probative value(and authenticity) of more direct forms of evidence

Class discussion. Safety evidence & standards

Consider modern safety standards such as IEC 61508 and DO-178. Whatis the ratio of direct evidence to backing evidence generated by theprocesses of these standards? Are they different? Are there differences inthe probative value? Should the ratio remain the same or differ withincreasing DAL/SIL?


Is this better evidence?

Flight test of overboard discharge from aircraft20 M.Squair (SSCS 2104) What should be in a safety case?: Evidence

Toulmin’s model of practical arguments

Current practices in formal safety argument are based on the practicalargument model [Toulmin 1958]

Focuses on the justification aspects of arguments rather than inferential.Argument parts consist of facts (evidence), conclusions, warrants, backingand qualifiers

The warrant is why it’s considered valid to move from the fact to theconclusion

The rebuttal is a legitimate constraint that may be placed on theconclusion drawn

Backing is evidence introduced if the warrant on the face of it is notcredible

The rebuttal part of Toulmin’s model is not found in current graphicalnotations such as GSN or CAE.

21 M.Squair (SSCS 2104) What should be in a safety case?: Argument

Toulmin’s model (cont’d)


A small philosophical quibble

The problem is that Toulmin developed his model so that one couldanalyse an argument, that is argument is used in the verb sense

Unfortunately in practice safety arguments tend to depart from LordCullen’s first point and inherently skew to an advocacy position

Thus the rebuttal part of Toulmin’s model gets overlooked, that is theword ’argument’ is used as a noun

From there it is a small step to the narrative fallacy e.g. presenting all thatgood data that the system is safe

Of course there’s very little evidence of rare catastrophic events becausethey’re, well, rare...


”The Nimrod safety case process was fatally undermined by a generalmalaise: a widespread assumption... that the Nimrod was ’safe anyway’(because it had successfully flow for 30 years) and the task of drawing upthe safety case became essentially a paperwork and ’tickbox’ exercise.”

The Nimrod Review Charles Haddon-Cave QC, 2009


Why should we believe the evidence and argument?

If you’re a lawyer answering this is your bread and butter

Evidence

Use of backing evidence to assure us of authentic artifacts, produced usinga credible process or method, by competent people using certified tools

Argument

Use of the ’right’ logic to reason about the right things:

Pascalian - reasoning about likelihood

Modal logic - reasoning about possibility

Dempster/Shafer (D/S) - reasoning about evidence

25 M.Squair (SSCS 2104) What should be in a safety case?: Arguing about belief

A little bit more about D/S logic

A mathematical theory of evidence[Shafer 1976]

Allows us to combine sources of evidence and deal with conflict, that’svery useful in safety arguments

We assess the degree of belief and plausibility of evidence

Combinatorial rules allow us to combine evidence from different sources

If there’s high conflict in the evidence it will show up as counterintuitiveresults...

26 M.Squair (SSCS 2104) What should be in a safety case?: Arguing about belief

Formal notations

Two graphical notations are available

Goal Structuring Notation (GSN). Developed by Kelly & others,there is a GSN community standard

Claims, Arguments, Evidence (CAE). Developed by Bishop &others, supported by Adelard’s Safety Case Editor tool

Both are graphical in nature to assist in clarity of argument

Both are based on Toulmin’s practical argument structure

Clarity does not denote soundness

The use of one particular notation or another does not infer any greater orlesser soundness upon the actual worth of the argument

27 M.Squair (SSCS 2104) Notations: Graphical techniques

Graphical notations for safety arguments

GSN versus CAE notation

28 M.Squair (SSCS 2104) Notations: Graphical techniques


2 Methodology


4 Notations





9 Conclusions

29 M.Squair (SSCS 2104) Producing a safety case:

Developing a safety argument in GSN

1 Establish top level goals (customer/statutory)

2 Record the stakeholders for the goals

3 Define derived requirements (standards, codes etc)

4 Establish (3) as goals (or constraints) and link to top goals

5 Break down the top level goals into sub-goals

6 Show how design & analysis decisions meet goals via strategies

7 Record the decisions as they are made

8 Justify strategies

Evidence versus argument

Evidence without argument is unexplained, argument without evidence isunfounded

30 M.Squair (SSCS 2104) Producing a safety case: Developing the argument

Example fragment of a safety argument in GSN notation


Class exercise: A safety case to remember


Dealing with scale and complexity

GSN has been extended in recent years to include:

Safety case modules. Allow the partitioning of cases into moreeasily managed modules and module interfaces (systems of systemsapproach)

Safety case patterns. Standardised templates to encourage re-use ofsuccessful arguments [Kelly, McDermid 1997]


Example modular safety case

Eurocontrol RVSM pre-implementation safety case


Example modular safety case (cont’d)

Eurocontrol RVSM Implementation module


Safety case patterns

Safety pattern: functional safety argument


Safety arguments as scientific hypothesis

The best tool that we have for differentiating between a good theory and abad one is the scientific method:

our hypothesis is that our system is safe

the argument is why we think this is justified

in science a justifiable hypothesis is not considered proven

in science the hypothesis is then challenged by others

but with safety argument is this (ever) the case?

The safety case as ’proof’ fallacy

An unchallenged safety case is essentially an appeal to authority argument,authority in this case being how impressive the report is

37 M.Squair (SSCS 2104) Producing a safety case: Challenging the safety case

So how do we challenge a safety case?

Four broad avenues of attack:

Deconstruction

Refutation

Disconfirming evidence

Or

disproof by construction i.e. an accident shows your safety argumentis flawed (not recommended)

The above might seem a lot but (for example) a claim that the likelihoodof a LOCA accident is 10−9 per reactor year is a very strong statement,and strong statements demand strong proof surely?


So how do we challenge a safety case?

Four broad avenues of attack:

Deconstruction

Refutation


Or disproof by construction i.e. an accident shows your safety argumentis flawed (not recommended)

The above might seem a lot but (for example) a claim that the likelihoodof a LOCA accident is 10−9 per reactor year is a very strong statement,and strong statements demand strong proof surely?


Deconstruction

Based on the work of french philosopher Jacque Derrida on the theory ofmeaning (and it’s inherent indeterminacy) and his use of it in critiquingphilosophical arguments [Armstrong, Paynter 2002]

Derrida’s view on arguments

An argument is defined by what it ignores and the perspectives it opposes(explicitly or implicitly)

Application to safety cases

Might a valid counter-argument, which would negate the parent, exist?

Deconstructionism is one way to test for the presence of acounter-argument


Deconstructionist technique

Develop a counter argument that seems warrantable and use this toexpose the internal flaws and contradictions in the original case

1 Reversal. Reverse the argument, ignore how warranted the original is& look for warrantable counter-arguments

2 Displacement. Compare the relative warrantedness of both3 Evaluate the three possible end states

I The original argument is found to need revisionI The counter argument is found to need revisionI They both turn out to be equally compelling1

1Due to the limits of deductive closure40 M.Squair (SSCS 2104) Producing a safety case: Challenging the safety case

Deconstruction (Class exercise)

Modelling software reliability

Argument. Software failures occur randomly because of the random nature ofinputs from the environment that trigger latent faults. Therefore we can applyclassical reliability techniques to software failures

What might be a warrantable counter argument, or arguments?


Refutation of argument [Greenwell et al. 2006]

Challenge the specific arguments on the basic of fallacious argumentstructures and refute them



Challenge the evidence with disconfirming evidence

Based on Karl Popper’s concept of the science project as one of trying todisconfirm theories not confirm them

Consider

Quality of the evidence provided (pool size, outlier handling, magicbullet approaches)

Hazard control coverage metrics (is the argument vulnerable)

Independence and dissimilarity of evidence sources

Then go out and gather strongly disconfirming evidence that targets thegaps



2 Methodology


4 Notations





9 Conclusions

44 M.Squair (SSCS 2104) Maintaining the safety case:

Safety case maintenance

In theory, a safety case should be maintained till system retirement

Example

The Long Term Safety Review of the U.Ks Magnox reactors, quoted in[Kelly 1998] found that lack of maintenance to the original safety case had causedit to become inconsistent with current plant design and operations. The reviewfurther found that adding to and re-evaluating a safety case that has become outof date with respect to current safety standards was problematic

In practice, unless significant effort is expended it rapidly falls out of date

Requires both regulatory & corporate buy in

Example

UK HSE originally required review (including HSE) of offshore safety cases at 3

yearly intervals, this was extended to 5 years in 2005


Safety case maintenance

One of the biggest challenges is maintaining the safety case in the face ofsystem changes

We would also like to use the safety case to assess changes for safetyimpact

We also have to repair the case after a change has been made, hopefully ina cost effective fashion

A graphical safety argument with traceability structures is invaluable forthese purposes [Kelly, McDermid 2001]



2 Methodology


4 Notations





9 Conclusions

47 M.Squair (SSCS 2104) Do safety cases work in the real world?:

Practical and theoretical problems with the approach

A number of of significant safety cases have been reviewed, and problemsfound with them

Magnox reactor safety review

Haddon enquiry into the Nimrod disaster

Ladkin analysis of the EUROCONTROL RVSM safety case

Knight analysis of Opalinus Clay Nuclear repository safety case

None of these were minor projects, so it appears that even when great careshould be taken, flawed arguments still appear

The theoretical problem is that for high consequence systems thelikelihood must be very, very low and we must have a very high faith in theargument that this is so. Do we?


A personal view, i.e. mine

Lord Cullen was wrong. We need to address the weaknesses beforewe can really take safety cases seriously

Safety cases cannot be written by an advocate alone (unavoidable biases)

Safety cases are a hypothesis (subject to adversarial disproof)

Caesers wife assessment is essential (squeaky clean independence)

There must be room for dissenting opinions (reinstate rebuttal)

Safety cases must be open (crowd sources adjunct reviews)

Parsimony of use (Only for significant societal concerns)

Names are important, call it a risk case (address framing effects)

It’s the evidence dummy! (apply Dempster-Schafer logic)

Safety cases should not be a burden (at least cost neutral)



2 Methodology


4 Notations





9 Conclusions

50 M.Squair (SSCS 2104) Limitations, advantages and disadvantages:

Limitations of the method

Limitations

Relies upon correspondence between safety argument and safety case

Relies upon peoples ability to reason and argue effectively, there’s nota lot of evidence that people are actually good at this


Advantages

Advantages are that

Is almost mandatory if working in a goal based regulatory environment

Is invaluable in organising the safety program documentation ’tail’

Can promote thought and discussion, if used appropriately

Can provide a change safety impact assessment capability in service


Disadvantages

Disadvantages:

Can become over time, another tick the box exercise

Is vulnerable to the narrative fallacy

Has a tendency to become an advocacy piece

Is very hard to review effectively without formal training

Can become an administrative burden that is perpetually chasing thesystem



2 Methodology


4 Notations





9 Conclusions

54 M.Squair (SSCS 2104) Conclusions:

Conclusions

Safety cases emerged out of the political and industrial landscape ofEngland in the late 1970’s, they reflect a particular societal viewpoint onboth who should be responsible for managing major hazards and how theyshould manage them.

They are in the end another tool, neither an end in themselves nordemonstrably the only way to assure the safety of complex systems.

Their current demonstrated deficiencies perhaps more demonstrate thedifficulty humans have in arguing rigorously and logically, than any specificlimitations of the method

55 M.Squair (SSCS 2104) Conclusions:

Bibliography

[Armstrong, Paynter 2002] Armstrong, J. M. and Paynter, S. P. (2002). Safe Systems:Construction, Destruction and Deconstruction. In: Redmill, F. and Anderson, T.(eds.), Current Issues In Safety Critical Systems, pp. 63-76, Springer-Verlag, Berlin.

[Bishop, Bloomfield 1998] Bishop, P. G. & Bloomfield, R. E. (1998). A Methodologyfor Safety Case Development. In: F. Redmill & T. Anderson (Eds.), IndustrialPerspectives of Safety-critical Systems: Proceedings of the Sixth Safety-criticalSystems Symposium, Birmingham 1998.

[DoD (US) 1993] DoD (US) (1993) Standard Practice for System Safety (1993) USDept of Defense Standard MIL-STD-882C, 19 January 1993.

[Greenwell et al. 2006] Greenwell, W. S, Holloway, M., C. Knight, J.C., (2006) ATaxonomy of Fallacies in System Safety Arguments, Proceedings of the 2006International System Safety Conference.

[Haddon-Cave 2009] Cave, C.H. (2006) An Independent Review Into the Broader IssuesSurrounding the Loss Of The RAF Nimrod MR2 Aircraft XV230 In Afghanistan in2006, The Stationary Office, Tech. Rep., 2006

56 M.Squair (SSCS 2104) Further reading:

[Hawkins et al., 2011] Hawkins, R., Kelly, T., Knight, J. and Graydon, P. (2011) A newapproach to creating clear safety arguments, in Proc. Safety Critical Systems Symp.,Feb. 2011.

[Kelly, McDermid 1997] Kelly T, McDermid J. (1997) Safety case construction andreuse using patterns. In: Proc. 16th Intl. Conf. Computer Safety, Reliability, andSecurity (SAFECOMP97). New York, 1997.

[Kelly 1998] Kelly, T.P., (1998) Arguing Safety, A Systematic Approach to ManagingSafety Cases, Doctoral Thesis, Dept of Computer Science, University of York 1998.

[Kelly, McDermid 2001] Kelly T, McDermid J. (2001) A systematic approach to safetycase maintenance. Reliability Engineering and System Safety 2001;71(3):271-284.

[MOD (UK) 2007] UK MoD (2007) Defence Standard 00-56 Issue 4: Safetymanagement requirements for defence systems, HMSO.

[Shafer 1976] Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton, NJ,Princeton University Press.

[Toulmin 1958] S. E. Toulmin, S.E., (1958) The Uses of Argument, CambridgeUniversity Press, 1958.

57 M.Squair (SSCS 2104) Further reading:

safety cases and arguments - a tutorial · pdf filesafety cases and arguments - a tutorial iet...

Documents