The development of a tool to predict team performance

Download The development of a tool to predict team performance

Post on 04-Sep-2016




1 download


<ul><li><p>am</p><p>, MougrougKinited</p><p>Keywords:MethodologyPerformance predictionSmall groups</p><p>cation</p><p>the process will be successful. The team may launch the missilesuccessfully, but it may still miss the target. Consequently, proba-bilities of team success should be convolved with probabilities ofprocess success to obtain a quantitative prediction of overall success.</p><p>1.1. Theoretical basis for the tool</p><p>All the constructs and ideas for the tool are already in existence.There are a number of reviews which describe the evolution of</p><p>* Corresponding author. Tel.: 44 7590 065250; fax: 44 1509 223940.E-mail addresses: (M.A. Sinclair), c.e.siemieniuch@</p><p> (C.E. Siemieniuch), (R.A. Haslam), (L. Evans).1 Tel.: 44 1509 635230; fax: 44 1509 635231.2 Tel.: 44 1509 223042; fax: 44 1509 223940.3</p><p>Contents lists availab</p><p>Applied Erg</p><p>journal homepage: www.els</p><p>Applied Ergonomics 43 (2012) 176e183Tel.: 44 117 302 predict the performance of teams when executing a process, inanswer to a direct request from engineers and human factorsexperts in the UK defence industry. In the version discussed in thispaper, the tool is entitled Performance Evaluation and Assessmentfor Teams, version 9.1 (PEAT 9.1).</p><p>The tool provides designers of military systems at the concep-tual stages of design (when variables are still variables and notparameters) some help in risk reduction exercises when consid-ering the stafng of processes. Some sample questions for whichthe tool could help in providing answers at this early stage are:</p><p>hood of success becomes unacceptable? By howmuch can the attributes of the individuals in the team bereduced, before the likelihood of success becomes unacceptable?</p><p>These questions are phrased in terms of success. This is a signi-cant point; success is denedhere as executing the process correctlyand attaining all of the goals of the process, with no reworking, noextra resources, no extra time involved. Likelihood of success isexpressed as a probability, as usual. Note that the tool predicts thesuccess of the team in executing the process; it does not predict that1. Introduction</p><p>We report the development, veri0003-6870/$ e see front matter 2011 Elsevier Ltddoi:10.1016/j.apergo.2011.05.004concepts are still uid, including the structure of the team(s) which are expected to be operators withinthe system. It enables answers to be calculated for questions such as What happens if I reduce teamsize? and Can I reduce the qualications necessary to execute this process and still achieve the requiredlevel of success?.The tool has undergone verication and validation; it predicts fairly well and shows promise. An</p><p>unexpected nding is that the tool creates a good a priori argument for signicant attention to HumanFactors Integration in systems projects. The simulations show that if a systems project takes full accountof human factors integration (selection, training, process design, interaction design, culture, etc.) then thelikelihood of team success will be in excess of 0.95. As the project derogates from this state, the likeli-hood of team success will drop as low as 0.05. If the team has good internal communications and goodindividuals in key roles, the likelihood of success rises towards 0.25. Even with a team comprising thebest individuals, p(success) will not be greater than 0.35.It is hoped that these results will be useful for human factors professionals involved in systems design.</p><p> 2011 Elsevier Ltd and The Ergonomics Society. All rights reserved.</p><p>and validation of a tool</p><p> What is the likelihood that this team will be successful inexecuting the process under consideration?</p><p> By how much can the team size be reduced, before the likeli-Accepted 2 May 2011 domains. It is expected to be used by systems engineers in initial stages of systems design, whenArticle history:Received 10 June 2010</p><p>The paper describes the development of a tool to predict quantitatively the success of a team whenexecuting a process. The tool was developed for the UK defence industry, though it may be useful in otherThe development of a tool to predict te</p><p>M.A. Sinclair a,*, C.E. Siemieniuch b,1, R.A. Haslam c,2</p><p>aCentre for Innovative &amp; Collaborative Engineering, Loughborough University, LoughborbDepartment of Electrical &amp; Electronic Engineering, Loughborough University, LoughbocDepartment of Ergonomics, Loughborough University, Loughborough LE11 3TU, UniteddHuman Factors Department, BAE Systems Advanced Technology Centre, BS34 7QW, Un</p><p>a r t i c l e i n f o a b s t r a c tand The Ergonomics Society. All riperformance</p><p>.J.d.C. Henshawb,1, L. Evans d,3</p><p>h LE11 3TU, United Kingdomh LE11 3TU, United KingdomgdomKingdom</p><p>le at ScienceDirect</p><p>onomics</p><p>evier .com/locate/apergoghts reserved.</p></li><li><p>knowledge about teams; good examples of these are (Cummingset al., 1977; Sundstrom et al., 1990; Barrick and Mount, 1991;Leonard and Freedman, 2000; Sundstrom et al., 2000; Devine andPhilips, 2001; Hare, 2003; Kozlowski and Ilgen, 2006; Stewart,2006), There has been steady progress in understanding teamsand teamwork, albeit in different schools of thought, but it isgenerally agreed that coherent, predictive models for practitionersand systems developers are few in number. However, there aremodels in the literature that could be adapted for predictivepurposes; for example Thurstones Five Factor Model (Thurstone,1934) and Salas et al.s Big Five model (Salas et al., 2005).</p><p>In both of these cases, and including PEAT, the models makevery limited use of the research discussed in the reviews above.This is illustrated by both (Benn, 2005) and (Shanahan, 2005), whoindependently have produced similar directed graphs combiningmuch of the research discussed in these reviews. Fig. 1 is fromShanahan, and illustrates an important point; it is immediatelyevident that no practical and useful predictive model could becreated from this work as there are toomany unquantied feedback</p><p>loops, and the input data requirements to drive the model areprohibitive. Consequently, a simplied model is necessary.</p><p>2. Development of the tool</p><p>2.1. The general approach</p><p>The development process for the tool followed standard engi-neering processes, as opposed to a scientic process. This wasdeliberate; science answers the question, why ., whereas engi-neering answers the question what solution can we nd ..Consequently, the approach adopted here is a version of the stan-dard systems engineering methods as outlined in Haskins (2010).Within this so-called VEE process, the stages from Detaileddesign to Integration and test were iterated to produce incre-mental improvements to the tool, so that the development processalso corresponded to a spiral development model (Boehm, 1988),resulting in PEAT 9.1, i.e. the version which is described in thispaper.</p><p>M.A. Sinclair et al. / Applied Ergonomics 43 (2012) 176e183 177Fig. 1. Illustration of variables affecting the performance of a team. From Shanahan (2005).</p></li><li><p>Three constraints guided the development of the tool. Thesewere dened in the initial phases of tool development based oninterviews with key stakeholders. Separate interviews wereundertaken with individuals involved in the engineering lifecycle;3 engineers, 2 project managers, 4 human factors experts, and 2senior engineering discipline managers, all within the samedefence organisation but in different business units, ranging fromaerospace to underwater:</p><p>A user-dened constraint: If the tool needs more that a two-page manual to explain it, it will not be used. This was a unan-imous view among a target group of engineers who wereinterviewed at the beginning of the project and reects thepracticalities of engineering life; rstly, there are never enoughengineers in society, and therefore they are usually highlyunwilling to spend signicant time on training for an unknowntool that might not produce time, cost or quality benets.A business process constraint: Systems designers are alreadyfamiliar with, and may be using as a standard procedure, tech-niques such as HEART (Human Error Assessment &amp; ReductionTechnique (Williams, 1986)) and CREAM (Cognitive Reliabilityand Error Analysis Method - (Hollnagel, 1998)) for assessing thereliability of individuals. The tool should incorporate thesetechniques, or enable the incorporation of any other equivalentcompany technique, in order to enhance the ease of acceptanceinto design processes, and to help to address the negativeaspects in the bullet-point above.A design constraint: At the conceptual stages of design, little willbe known about the individuals in the team that will beexpected to execute any new processes, apart from genericattributes. Equally, the process will be undened e perhaps just</p><p>a single ow diagram sketched on a sheet of paper. Hence, thetool must make a minimum demand for input data by reducingto a minimum the variables involved.</p><p>In summary, the rst constraint has been met, in the form ofa two-page user manual, though a longer version of the manual isalso available. The second constraint has been met by creatinga three-stage tool: Stage 1 collects data about the attributes of theindividuals in the team and the intercommunications deemednecessary for execution of the process; Stage 2 collects an analysisof the process organisational environment, using either HEART orCREAM, or the organisations own in-house technique for thispurpose; and Stage 3 convolves the outputs of Stages 1 and 2, anddelivers a likelihood of success, depending on the binding of theteam to the process. Fig. 2 illustrates the three stages for using thetool.</p><p>The third constraint has beenmet by reducing to aminimum thevariables involved, and hence the input data required.</p><p>Based on the constraints described above, the model shouldcontain only those variables widely discussed in the literature, andwhich practitioners deem to be critical to team performance.Because of its intended use by engineers and others not necessarilyexperts in psychological matters, only the irreducible number ofvariables should be used. These were:</p><p> Trustworthiness (dependability of an individual to deliverresults, on time, and in full).</p><p> Teamworking skills (how constructive the person is in aidingthe team to its goals).</p><p> Domain knowledge and skills possessed by the individual forthe process under consideration.</p><p>M.A. Sinclair et al. / Applied Ergonomics 43 (2012) 176e183178Fig. 2. Stages of PEAT 9.1, illustrating capture of the team and binding to the process, theadjustment of these due to teamworking and teamworker characteristics, and nal convolucalculation of initial Human Error probabilities (HEPs) using standard techniques, thetion to deliver a prediction of success.</p></li><li><p>Ergo Communication links between team members necessitated bythe process.</p><p> Authority relationships within the team, as determined byrank, role, and/or expertise.</p><p>The derivation process for this choice of variables was by a seriesof 5 discussions among a group of academic subject matter experts(seven in total, with over 100 years of experience in working withindustry) allied to the reviews mentioned earlier.</p><p>Hence, we have another ve-factor model which has someoverlapswith those of (Thurstone,1934) Salas et al.s Big Fivemodel(Salas et al., 2005). These variables selected for PEAT are those withwhich systems developers and systems operators are familiar on aneveryday basis, thus adding to the acceptability of the model.</p><p>2.2. Description of the tool</p><p>The tool currently exists as a spreadsheet, with individualworksheets for each of the stages 1e3 described above.</p><p>For each of the rst three variables listed at the end of thesection above (Trustworthiness, Teamworking skills and Domainknowledge and skills) four-point rating scales are provided, wordedfor non-experts. Each runs from a zero point to expert. Commu-nications are captured in a matrix as one-way links; a discussionbetween two team members is represented by two links. There isno attempt to characterise the link itself.</p><p>Authority within the team is assessed by a ve-point scale. Thisruns from instructed, obeys through discusses as equals (themid-point) to instructs, expects obedience. The rationale for thesechoices was ease-of-use by non-experts; the verication and vali-dation studies reported later indicate that the format is sufcientfor purpose.</p><p>In stage 1, each team member is rated on these variables. Therst three ratings for trustworthiness, teamworking and knowledgeare combined to produce a Performance Shaping Factor (PSF),which acts as a multiplier that adjusts each individuals propensityfor error on the process, used later in the assessment. The PSF isobtained by means of a look-up table. Since the quality of this tableis critical to the predictions of the tool, its construction is outlinedbelow.</p><p>The look-up table was constructed by a total of 8 human factorsexperts, all practitioners with long experience of teams, organisa-tions and processes and with professional qualications in cognatedisciplines and with professional practitioner afliations. In allthese experts represented some 200 years of professional experi-ence in industry.</p><p>For each, this was a time-consuming, difcult task; as one expertexpressed it, So, you want me to produce a PSF for some individualwith a trust rating of, say, 2, a teamworking rating of 1, anda knowledge rating of 2, for some process I do not know? Theresponse was Exactly that, O Guru. And they did; the results weresmoothed and inserted into the tool. It is to the credit of theseexperts that PEAT can predict satisfactorily as shown by the veri-cations and validations.</p><p>The next step, stage 2, captures the organisational environmentaround the process. This corresponds to standard Human ReliabilityAssessment approaches (e.g. (Swain and Guttmann, 1983; Embreyet al., 1984; Kirwan, 1988; Lee et al., 1988; Dougherty, 1990;Kirwan, 1994; Cooper et al., 1996; Hollnagel, 1996; Perrow, 1999;Bot, 2003; Dekker, 2005a,b; Hollnagel, 2005; Kirwan, 2005;Kirwan and Gibson, 2007; Johnson et al., 2009)). Of these, in theUK the techniques known as HEART (Human Error Assessment &amp;Reduction Technique) (Williams, 1986) and CREAM (CognitiveReliability and Error Analysis Method) (Hollnagel, 1998) are well-</p><p>M.A. Sinclair et al. / Appliedknown and widely used. Therefore, both HEART and CREAM wereincorporated into the tool, though organisations which have theirown methods for assessing human reliability could replace these.</p><p>The nal stage 3 combines the base probability of error fromHEART or CREAM, the PSF values for each individual (adjusting thebase probability of error according to each individuals charac-terisics), the authority ratings, and the communications matrix (toaccount for the contributions of other team members to ones ownperformance) to arrive rstly at what is called interactive proba-bilities of error for each individual; in other words, acknowl-edging peer effect. The initial probabilities of error calculated byHEART, CREAM (or other technique) are ad...</p></li></ul>


View more >