human reliability assessment
DESCRIPTION
Human Reliability Assessment. Grace Kennedy [email protected] 16 th /19 th October 2006 06ELD061/06ELP461. Objectives for the Sessions. Understand the Human Reliability Assessment process Gain practical experience of a simple task analysis - PowerPoint PPT PresentationTRANSCRIPT
Human Reliability Assessment
Grace Kennedy
16th/19th October 200606ELD061/06ELP461
Objectives for the Sessions
• Understand the Human Reliability Assessment process
• Gain practical experience of a simple task analysis
• Gain practical experience of error identification
• Gain practical experience of representation
• Gain practical experience of quantifying error probabilities in a simple example
Human Error Example 1
• A KC-135 Aircraft was being pressurised at ground level.
• The outflow valves were capped off during a 5 year overhaul and never re-opened.
• A civilian depot technician was using a home-made gauge, and no procedure.
• The technician's gauge didn't have a max "peg" for the needle which had gone round the gauge more than once.
• The result….
Human Error Example
Human Error Example 2
Helios Crash 2005
Extract taken from BBC News Site
http://news.bbc.co.uk/1/hi/world/europe/6036507.stm?ls
• Pilot misread instruments AND misinterpreted warning signals
• Maintenance left pressure control in wrong setting
• Manufacturer did not respond adequately to previous similar incidents
Predicting errors
• Task analysis and error identification
Preventing errors
• Specifying training requirements
• Equipment design (e.g. pressure gauge)
• Detailed procedures (administrative control)
Ultimately:
• Reduce risk
• Save money
• Justify design decisions
What can be done about it?
“For every $1 spent in the early stage,
approximately $10,000 are saved (if the problem
were to be fixed later instead).” – Manprint
Understanding Human Error
Process Techniques
Accident Analyses
Prediction
Risk Assessment
Psychology
Environment HCI Design
What is HRA?
HRA = Human Reliability Assessment
HRA Process Outline
• Task analysis is used to describe and understand the human interactions with the system
• The results of the task analysis are used with an error taxonomy (classification scheme) to allow error identification
• The identified errors are analysed either qualitatively or quantitatively
• The process is repeated each time a design iteration occurs
Human Reliability Assessment Process
General HRA Process – Kirwan, 1994
Human Reliability Assessment Process
• Problem Definition
• Task Analysis– Describe what is done
– Improve analyst’s knowledge
• Error Identification– Taxonomy
– Failure criteria
• Representation– Fault tree/event tree
– Risk model
• Quantification– e.g. HEART
• Impact Assessment– Effect of errors
– Risk contribution
• Error Reduction– Re-design tasks
– Add engineered features
– Procedures / training
• Quality Assurance– Appropriate techniques
– Technical checking
• Documentation
Definitions of Terms
• Process: The overall HRA process
• Method: The steps in a process
– e.g. Task Analysis, Quantification, etc
• Technique: The specific implementation of a method(s)
– e.g. HEART, THERP, etc
• Tool: A software tool to record and guide the use of a technique
– e.g. Fault Tree +
HRA Techniques
• Many HRA techniques available
• Working to different levels of detail on different concepts
• From expert judgement techniques (e.g. APJ, PC)
• Hazard identification techniques (e.g. HAZOPS, THEA)
• To quantitative techniques (e.g. HEART, THERP)
• To second generation techniques (e.g. CREAM, ATHEANA)
Human Reliability Assessment Process
General HRA Process – Kirwan, 1994
TASK ANALYSIS
Task Analysis
• Range of techniques to understand what humans are
required to do in order to achieve a system goal
– Collect and organise information
– Improve the analyst’s understanding
– Structured approach
– Support to design and assessment
A Guide to Task Analysis,
Barry Kirwan & Les Ainsworth (1992),
Taylor and Francis
ISBN 07484-0058-3
Hierarchical Task Analysis
• Expresses a job or function in terms of goals,
operations and plans
– Goals Objectives to be achieved
– Operations Actions required to achieve the goals
– Plans Conditions under which the actions are
carried out
Hierarchical Task Analysis Example
• Express the task of making a cup of tea using HTA
– Goals Objectives to be achieved (e.g. Make Tea)
– Operations Actions required to achieve the goals (e.g.
Boil water, Add milk / sugar)
– Plans Conditions under which the actions are
carried out (e.g. boil the water before
adding it to the cup)
Example provided using TaskArchitect software
Making Tea - one solution (1 of 5)
Bar beneath the activity shows no further development
Stub beneath the activity shows further development has taken place
Plan describes the logic
Making Tea - one solution (2 of 5)
Making Tea - one solution (3 of 5)
Making Tea - one solution (4 of 5)
Making Tea - one solution (5 of 5)
Hierarchical Task Analysis - Practical
• Express the task of fitting an electric plug using HTA
– Goals Objectives to be achieved (e.g. Fit plug)
– Operations Actions required to achieve the goals (e.g.
Strip outer casing, Twist exposed
wires)
– Plans Conditions under which the actions are
carried out (e.g. fit the fuse before
closing up the plug to the
cup)
Hierarchical Task Analysis - Guidance
• State an overall goal (box at the top)
• Breakdown each goal or sub-goal one at a time (i.e.
finish one box before moving on)
• Ensure all the actions under a goal are relevant and
would actually achieve the stated goal
• Keep the order and logic in the plans (make the plans
specific to the goal)
Work in Pairs - 10 minutesA solution on pink sheet
Wiring a plug – HTA one solution (1 of 4)
Wiring a plug – HTA one solution (2 of 4)
Wiring a plug – HTA one solution (3 of 4)
Wiring a plug – HTA one solution (4 of 4)
Human Reliability Assessment Process
General HRA Process – Kirwan, 1994
ERROR IDENTIFICATION
Error Identification - General
• Task Analysis describes the activities necessary to achieve a goal
• An Error Taxonomy (classification scheme) can be used to identify specific errors
• Many errors will be possible, so need to understand– Error effects (relating to the task goal)
– Failure criteria (goal failure)
• Produce a list of identified errors, which lead to goal failure
• Organise the information in a Tabular Task Analysis
Error Identification - Tabular Task Analysis
• Use the information from the HTA
• Create a Tabular Task Analysis (TTA)
• Error taxonomy (classification scheme) to identify errors
• Understand
– Error effects
– Failure criteria
• List of identified errors
Tabular Listing from HTAID Task Plan0 Wire an electric plug do in sequence 1-50.1 Collect tools0.2 Unscrew plug cover0.3 Prepare lead do in sequence 1-60.3.1 Estimate length of stripped wire required to reach earth terminal0.3.2 Strip outer casing according to estimate
0.3.3Check yellow/green wire reaches earth terminal whilst outer casing exceeds holder by 5 mm
0.3.4 Cut blue and brown wires to reach their terminals0.3.5 Strip each of the coloured leads to leave exposed wire0.3.6 Twist exposed wires on each coloured lead
0.4 Ensure correct fuse is in place
do in sequence 1-3; If mismatch between required and in-situ fuse then do ( 4)
0.4.1 Locate appropriate instructions for equipment0.4.2 Read fuse requirement0.4.3 Compare fuse requirement with given fuse0.4.4 Change the fuse do in sequence 1-30.4.4.1 Select the correct fuse0.4.4.2 Extract fuse from plug0.4.4.3 Insert correct fuse0.5 Attach plug to lead do in sequence 1-40.5.1 Thread lead through holder0.5.2 Fit each twisted wire to correct terminal0.5.2.1 Select one coloured lead0.5.2.2 Identify lead based on colour (earth, live, neutral)0.5.2.3 Locate correct terminal for selected lead0.5.2.4 Unscrew terminal0.5.2.5 Place exposed twisted wire through terminal hole0.5.2.6 Tighten terminal screw0.5.3 Secure main lead holder0.5.4 Replace plug cover
TTA Example – Selected ActivitiesID Task Plan Error
IDError Type Immediate
effects of errorDetection of error
Recovery of error
0.4 Ensure correct fuse is in place do in sequence 1-3; If mismatch between required and in-situ fuse then do ( 4)
0.4.1 Locate appropriate instructions for equipment
0.4.2 Read fuse requirement
0.4.3 Compare fuse requirement with given fuse
0.4.4 Change the fuse do in sequence 1-3
0.4.4.1 Select the correct fuse
0.4.4.2 Extract fuse from plug
0.4.4.3 Insert correct fuse
Error Taxonomy
• Classification scheme
• Generic error types
• Similar to HAZOP guidewords
• Taxonomy can be made domain specific
Error Taxonomy – SHERPA (see handout)
• Example error types for an action task
– E3 Action Omitted
– E4 Action too much
– E5 Action too little
– E9 Right action wrong object
– E10 Wrong action right object
Tabular Task Analysis Example - PracticalID Task Plan Error
IDError Type Immediate
effects of errorDetection of error
Recovery of error
0.4 Ensure correct fuse is in place do in sequence 1-3; If mismatch between required and in-0.4.1 Locate appropriate instructions for
equipmentE3 Action omitted Instructions not
obtainedUnable to confirm fuse type
Re-start task with instructions
E16 Wrong information obtained
Instructions for another device obtained
May not detect
Re-start task with instructions
0.4.2 Read fuse requirement
0.4.3 Compare fuse requirement with given fuse
0.4.4 Change the fuse do in sequence 1-3
0.4.4.1 Select the correct fuse
0.4.4.2 Extract fuse from plug
0.4.4.3 Insert correct fuse
Tabular Task Analysis - Guidance
• See Handout (blank TTA)
• Review each activity one at a time
• Read through the generic errors in SHERPA
• Add error types to the TTA and fill-in the remaining
columns (see example for guidance)
Work in Groups (max. 5) - 10 minutes
Tabular Task Analysis – Solution to Practical
• See handout sheet for example error types against each activity (example is not a comprehensive record)
ID Task Plan Error ID
Error Type Immediate effects of error
Detection of error
Recovery of error
0.4 Ensure correct fuse is in place
do in sequence 1-3; If mismatch between required and in-situ fuse then do ( 4)
0.4.1 Locate appropriate instructions for equipment
E3 Action omitted Instructions not obtained
Unable to confirm fuse type
Re-start task with instructions
E16 Wrong information obtained
Instructions for another device obtained
May not detect
Re-start task with instructions
0.4.2 Read fuse requirement
E3 Action omitted Fuse type not obtained
Unable to complete step 0.4.3
Return to instructions
E16 Wrong information obtained
Incorrect fuse information used
May not detect - device could fail
Repeat task with correct information
0.4.3 Compare fuse requirement with given fuse
E11 Check omitted Incorrect in-situ fuse not detected
May not detect - device could fail
Repeat task
E16 Wrong information obtained
Incorrect in-situ fuse not detected
May not detect - device could fail
Repeat task
0.4.4 Change the fuse
do in sequence 1-3
0.4.4.1 Select the correct fuse
E3 Action omitted No replacement fuse
Unable to complete step 0.4.4.3
Repeat task
E9 Right action on wrong object
Incorrect fuse selected
May not detect - device could fail
Repeat task
0.4.4.2 Extract fuse from plug
E3 Action omitted Original fuse left in place
Unable to complete step 0.4.4.3
Repeat task
0.4.4.3 Insert correct fuse
E3 Action omitted No fuse fitted Device does not work
Repeat task
E17 Misalign Fuse fitted incorrectly
Device does not work
Repeat task
Example TTA - Note this TTA is not comprehensive as additional error types may apply
You cannot read this, but . . .see green handout
Error Identification - Question
• Assume the fuse must be changed
• Review the tasks to achieve the goal at 0.4
• Use the error taxonomy (SHERPA) to identify :
– Example of an error leading to no fuse being fitted
– Example of an error leading to the incorrect fuse being fitted
• Remember
– Error Effects
– Failure Criteria
Work in pairs - 5 minutes
Error Identification Question - One Solution
• Errors leading to no fuse being fitted
– Step 0.4.4.3, Error E3 Action omitted
• Errors leading to the incorrect fuse being fitted
– Step 0.4.3, Error E11 Check omitted
– Step 0.4.4.1, Error E9 Right action on wrong object
Human Reliability Assessment Process
General HRA Process – Kirwan, 1994
Representation
Human Reliability Assessment and Risk Models
• Risk models will usually include human errors for
quantification (human as mitigation)
• Human Reliability Assessor will collaborate with the
Risk Modeller
– Further investigation may be needed in order to carry out
Human Reliability Assessment
– Additional errors may be identified for inclusion in the risk
model
– Changes to models may be necessary to represent human
error
Risk Assessment - General
• Risk = Frequency x Consequence/Severity
• Assessment of a complex system requires a structured process (Probabilistic Safety Assessment)
• Operation of the system is represented by a model (risk model)
• Risk model represents features in the system that prevent or mitigate against serious consequences (e.g. safety systems, intervention from human operators)
Risk Models – A Whirlwind Tour
• Hazard identification process used to establish a set of initiating events (what can happen to the system)
• Frequency of each initiator is assessed
• Consider the effects of each initiator on the system
• Typically use event trees to model accident sequences
• System features are ‘modelled’ as events in an Event Tree (ask success/failure questions as top events)
• Fault trees used to investigate detailed causes of equipment/system/human failure
An Event Tree
S
F
Respond to Alarm
Shut Valve
Start Pump
Success
Success (recovered)
Failure 1
Failure 2
Failure probability = x Success probability = 1 -x
x
1 - xInitiating event – system
leak
An Event Tree - Quantified
S
F
Respond to Alarm
Shut Valve
Start Pump
S1
S2
F1
F2
P(F) = F1 + F2 = (0.999 x 0.01 x 0.01) + 0.001 = 0.0011
P(S) = 1 – Failure = 1 – 0.0011 = 0.9989
0.001
0.999Initiating event – system
leak
0.01
0.01
0.99
0.99
Fault Tree+ from Isograph(http://isograph-software.com/ftpovereta.htm)
A Fault Tree
Valve Fails to Shut
Electrical signal to
valve fails
Mechanical valve failure
Operator fails to demand valve to
shut
OR
A B C
Failure probability = A + B + C – AB – AC – BC + ABC
A
CB
For OR use A U B U C
For AND use A n B n C
Fault Tree - Practical Example
Create a Fault Tree for incorrect fuse in place (i.e. 0.4)
• Two types of boolean operators
1. OR
Occurrence of ANY event below causes failure above
2. AND
Only the occurrence of ALL events below causes failure above
OR
AND
Fault Tree - Practical Example Guidance
• Use the errors identified as the branches to the trees
• Think about the HTA to give an indication of the layers required
• Think about which operator to use
Work in Groups (max. 5) - 10 minutesA solution on blue sheet
Fault Tree - A Solution
Incorrect fuse is in place
Appropriate instructions not
found
Fuse requirements not read
Fuse requirement not compared with
given fuse
Fuse not changed
Correct fuse not selected
Fuse not extracted from plug
Correct fuse not inserted
OR
OR
Human Reliability Assessment Process
General HRA Process – Kirwan, 1994
QUANTIFICATION
Human Error Probabilities - Quantification
• Human Reliability Assessment exists to provide quantification of the probability of human error
• Human Error Probabilities are used in Probabilistic Safety Assessment (risk models)
• Obtaining or generating Human Error Probabilities requires a range of techniques
Ways to get Quantitative Data
• Historical records
• Collected data (direct or simulated)
• Estimation techniques (constructive, comparative)
• Judgement and experience
Historical Records / Collected Data
• Number of recorded events of interest over time provides frequency of error
• Number of recorded events of interest over number of chances for event to occur provides the probability of error
• Strengths
– specific to the error of interest
– data validity (true values)
• Weaknesses
– may not have recorded all instances of error (under estimate)
– may need a lot of data to get a fair answer
– hard to identify root of some errors
– collection method may affect reliability
– collection in simulators may not be realistic for actual errors
– design changes over time may affect reliability
A Selection of Estimation Techniques
• Technique for Human Error Rate Prediction (THERP)
• Human Error Assessment and Reduction Technique (HEART)
• Success Likelihood Index Methodology (SLIM)
• Paired Comparisons (PC)
Estimation – THERP (1)
• Technique for Human Error Rate Prediction (NUREG/CR-1278, 1983)
• Collected data from civil PWRs in the USA (mainly control room actions, some manual valve actions)
• Presented as a database of Human Error Probabilities
• Flowchart of options to navigate the database
• Simple error taxonomy (omission, commission)
• Includes human error dependence model
• Construct HEPs from basic error data
Estimation - THERP (2)
• Strengths
– Powerful method with good auditability and supporting qualitative material
– Well suited to proceduralised, structured assessments
• Weaknesses
– Resource intensive (better with more experience)
– Not adaptive
– Limited error reduction information
Estimation - HEART (1)
• Human Error Assessment and Reduction Technique (Williams, 1990)
• Generic Task (GT) data in 9 categories
• List of 38 Error Producing Conditions (EPC)
• Select GT based on descriptions and examples (each task has a reliability attached)
• Modify base reliability by considering EPCs
• Advice included on possible error reduction measures
Estimation - HEART (2)
• Strengths
– Simple method, not resource intensive
– Error reduction suggestions
– Versatile (generic nature adapts to many tasks)
• Weaknesses
– Does not model dependence
– Results can vary greatly dependent on initial assessment (GT selection)
HEART – PC Demo
http://www.ewe.ch/regional/regional_2_6.html
Estimation - SLIM (1)
• Success Likelihood Index Method (Embrey et al, 1984)
• Panel of assessors (including subject experts)
• Evaluate performance shaping factors (PSFs) for task of interest
• Assign weighting as to the relative importance of the PSF to each other. Assign rating based on how useful the PSF is for the task.
• SLI derived from the sum of the ratings and weightings (ranks the errors)
• Calibrate SLIs with known data to convert SLI to HEP
P4P3P1 P2
Known data points
SLI
Estimation - SLIM (2)
• Strengths
– Flexible technique, good theoretical method
– Does not need task decomposition (task analysis and error taxonomies)
• Weaknesses
– Complex method, resource intensive
– Lack of valid calibration data (known values)
Estimation - Paired Comparisons (1)
• Paired comparisons (Hunns, 1982)
• Panel of assessors (including subject experts)
• Each assessor compares all possible pairs of error descriptions (decide which of the two is more likely for each pair)
• Combine all comparisons made by all assessors to produce a relative scaling of error likelihood
• Calibrate scaling with known data to convert to HEPs
Estimation - Paired Comparisons (2)
• Strengths
– Simple method, best use of limited known data
– Provides information on the quality of data from individual assessors (internal consistency check)
• Weaknesses
– Comparisons not suited to complex tasks
– Only suited to comparisons of similar tasks
Judgement / Experience
• Expert judgement
Judgement/Experience – APJ (1)
• Absolute Probability Judgement
• Panel of experts to provide direct generation of HEPs (subject experts and HRA expert)
• Assumes assessors are capable of making such estimates of reliability
• Describe the tasks of interest
• Describe errors and estimate HEPs
– Individual estimates aggregated
– Group consensus of estimates
Judgement / Experience - APJ (2)
• Strengths
– Simple method, allows constructive qualitative discussion
– Practical error reduction measures can be discussed during the assessment
• Weaknesses
– Prone to biases, may have little face validity
– Needs experienced experts
Criteria for Quantification Technique Selection
• Availability of data
• Applicability of data
• Ease of use (time, cost, resources, information)
• Data validity (justification)
• Experience of assessor
• Level of assessment needed
HEART - Practical Example
Use HEART to estimate the probability of fitting an
electric plug to a device incorrectly
• Assume :
– Time shortage for the task
– Written procedure (<10 steps) is used
– Over the shoulder checking is carried out
HEART – See Handout
• HEART paper
• Generic task descriptions
• Failure probabilities for each task description
– (50th percentile value should be used, 5th and 95th percentiles
indicate the uncertainty/range)
• Error producing conditions and associated
multiplication factors
HEART - Practical Example Guidance
• Read the Generic Task Descriptions
• Consider the task complexity and difficulty by examining the identified errors from the TTA
• Select a Generic Task
• Review the EPCs and select the ones you believe are relevant
• Modify the Generic Task base HEP using factors for selected EPCs
– See the worked example in the HEART paper
– EPC equation developed to avoid negative probabilities
Work in Groups (max. 5) - 10 minutesA solution on yellow sheet
HEART Practical - One (simple) Solution
• Probability of fitting an electric plug to a device incorrectly
– Generic Task F 0.003
– EPC 2 time shortage, x 11
• HEP = (0.003)(11) = 3.3E-2
Quantification Summary
• A range of HRA techniques is available
• Technique selection depends on the nature of the assessment
• Human Reliability Data can be difficult to obtain
• Human Reliability Data can be uncertain (range of probabilities)
• Information from the task analysis can be organised to suit the quantification technique (e.g. describe activities with HEART generic tasks in mind)
• Quantification must be based on detailed qualitative understanding
Capability Management Process for HRA
Prospective Analysis
HRA (concept)
HRA (design)
HRA (validate)
Deliverables required
Process Acceptance
CUSTOMER
Liaison
Market watch
SAFETY TEAMOwn, Maintain, Manage HRA Processes
Retrospective Analysis PRODUCT
Benchmarking
External SourcesOther
techniques/ external
databases/ review/researc
h
certify
accident/incident
Design
HRA Data
Safety Assessment Reporting
Experience/ Lessons Learnt
Maturity
Product design &
development process
BU REPSBest
practice, peer
review
Knowledge CaptureReporting and debriefing
FeedbackHE assessors, safety team
MeasuresCriteria (e.g. time, labour
involved, etc.)
Training
Process
DatabaseHRA data, lessons learnt, assessments, techniques
HRSelect & deploy team
Recording mgt.
Data & process
Process mgt.
Refine, tailor, what
& when
System Requirements
Regulatory Authorities
Human Reliability Assessment - Summary
• Understand the actions being investigated
• Use a structured approach to investigate and represent
the actions (task analysis)
• Consider the level of detail needed (compare with
description detail of available data)
• Understand the failure criteria
• Select an appropriate technique(s)
• Represent the identified errors in a risk model
Objectives – Re-visited
• Understand the Human Reliability Assessment process
• Gain practical experience of a simple task analysis
• Gain practical experience of error identification
• Gain practical experience of risk models
• Gain practical experience of quantifying error probabilities in a simple example
Useful References
• The following books provide a comprehensive guide to
Human Reliability Assessment and Task Analysis
A Guide to Practical Human Reliability Assessment,
Barry Kirwan (1994),
Taylor and Francis
ISBN 07484-0111-3
A Guide to Task Analysis,
Barry Kirwan & Les Ainsworth (1992),
Taylor and Francis
ISBN 07484-0058-3