the development of a tool to predict team performance

Download The development of a tool to predict team performance

Post on 04-Sep-2016




1 download

Embed Size (px)


  • am

    , MougrougKinited

    Keywords:MethodologyPerformance predictionSmall groups


    the process will be successful. The team may launch the missilesuccessfully, but it may still miss the target. Consequently, proba-bilities of team success should be convolved with probabilities ofprocess success to obtain a quantitative prediction of overall success.

    1.1. Theoretical basis for the tool

    All the constructs and ideas for the tool are already in existence.There are a number of reviews which describe the evolution of

    * Corresponding author. Tel.: 44 7590 065250; fax: 44 1509 223940.E-mail addresses: (M.A. Sinclair), c.e.siemieniuch@ (C.E. Siemieniuch), (R.A. Haslam), (L. Evans).1 Tel.: 44 1509 635230; fax: 44 1509 635231.2 Tel.: 44 1509 223042; fax: 44 1509 223940.3

    Contents lists availab

    Applied Erg

    journal homepage: www.els

    Applied Ergonomics 43 (2012) 176e183Tel.: 44 117 302 predict the performance of teams when executing a process, inanswer to a direct request from engineers and human factorsexperts in the UK defence industry. In the version discussed in thispaper, the tool is entitled Performance Evaluation and Assessmentfor Teams, version 9.1 (PEAT 9.1).

    The tool provides designers of military systems at the concep-tual stages of design (when variables are still variables and notparameters) some help in risk reduction exercises when consid-ering the stafng of processes. Some sample questions for whichthe tool could help in providing answers at this early stage are:

    hood of success becomes unacceptable? By howmuch can the attributes of the individuals in the team bereduced, before the likelihood of success becomes unacceptable?

    These questions are phrased in terms of success. This is a signi-cant point; success is denedhere as executing the process correctlyand attaining all of the goals of the process, with no reworking, noextra resources, no extra time involved. Likelihood of success isexpressed as a probability, as usual. Note that the tool predicts thesuccess of the team in executing the process; it does not predict that1. Introduction

    We report the development, veri0003-6870/$ e see front matter 2011 Elsevier Ltddoi:10.1016/j.apergo.2011.05.004concepts are still uid, including the structure of the team(s) which are expected to be operators withinthe system. It enables answers to be calculated for questions such as What happens if I reduce teamsize? and Can I reduce the qualications necessary to execute this process and still achieve the requiredlevel of success?.The tool has undergone verication and validation; it predicts fairly well and shows promise. An

    unexpected nding is that the tool creates a good a priori argument for signicant attention to HumanFactors Integration in systems projects. The simulations show that if a systems project takes full accountof human factors integration (selection, training, process design, interaction design, culture, etc.) then thelikelihood of team success will be in excess of 0.95. As the project derogates from this state, the likeli-hood of team success will drop as low as 0.05. If the team has good internal communications and goodindividuals in key roles, the likelihood of success rises towards 0.25. Even with a team comprising thebest individuals, p(success) will not be greater than 0.35.It is hoped that these results will be useful for human factors professionals involved in systems design.

    2011 Elsevier Ltd and The Ergonomics Society. All rights reserved.

    and validation of a tool

    What is the likelihood that this team will be successful inexecuting the process under consideration?

    By how much can the team size be reduced, before the likeli-Accepted 2 May 2011 domains. It is expected to be used by systems engineers in initial stages of systems design, whenArticle history:Received 10 June 2010

    The paper describes the development of a tool to predict quantitatively the success of a team whenexecuting a process. The tool was developed for the UK defence industry, though it may be useful in otherThe development of a tool to predict te

    M.A. Sinclair a,*, C.E. Siemieniuch b,1, R.A. Haslam c,2

    aCentre for Innovative & Collaborative Engineering, Loughborough University, LoughborbDepartment of Electrical & Electronic Engineering, Loughborough University, LoughbocDepartment of Ergonomics, Loughborough University, Loughborough LE11 3TU, UniteddHuman Factors Department, BAE Systems Advanced Technology Centre, BS34 7QW, Un

    a r t i c l e i n f o a b s t r a c tand The Ergonomics Society. All riperformance

    .J.d.C. Henshawb,1, L. Evans d,3

    h LE11 3TU, United Kingdomh LE11 3TU, United KingdomgdomKingdom

    le at ScienceDirect


    evier .com/locate/apergoghts reserved.

  • knowledge about teams; good examples of these are (Cummingset al., 1977; Sundstrom et al., 1990; Barrick and Mount, 1991;Leonard and Freedman, 2000; Sundstrom et al., 2000; Devine andPhilips, 2001; Hare, 2003; Kozlowski and Ilgen, 2006; Stewart,2006), There has been steady progress in understanding teamsand teamwork, albeit in different schools of thought, but it isgenerally agreed that coherent, predictive models for practitionersand systems developers are few in number. However, there aremodels in the literature that could be adapted for predictivepurposes; for example Thurstones Five Factor Model (Thurstone,1934) and Salas et al.s Big Five model (Salas et al., 2005).

    In both of these cases, and including PEAT, the models makevery limited use of the research discussed in the reviews above.This is illustrated by both (Benn, 2005) and (Shanahan, 2005), whoindependently have produced similar directed graphs combiningmuch of the research discussed in these reviews. Fig. 1 is fromShanahan, and illustrates an important point; it is immediatelyevident that no practical and useful predictive model could becreated from this work as there are toomany unquantied feedback

    loops, and the input data requirements to drive the model areprohibitive. Consequently, a simplied model is necessary.

    2. Development of the tool

    2.1. The general approach

    The development process for the tool followed standard engi-neering processes, as opposed to a scientic process. This wasdeliberate; science answers the question, why ., whereas engi-neering answers the question what solution can we nd ..Consequently, the approach adopted here is a version of the stan-dard systems engineering methods as outlined in Haskins (2010).Within this so-called VEE process, the stages from Detaileddesign to Integration and test were iterated to produce incre-mental improvements to the tool, so that the development processalso corresponded to a spiral development model (Boehm, 1988),resulting in PEAT 9.1, i.e. the version which is described in thispaper.

    M.A. Sinclair et al. / Applied Ergonomics 43 (2012) 176e183 177Fig. 1. Illustration of variables affecting the performance of a team. From Shanahan (2005).

  • Three constraints guided the development of the tool. Thesewere dened in the initial phases of tool development based oninterviews with key stakeholders. Separate interviews wereundertaken with individuals involved in the engineering lifecycle;3 engineers, 2 project managers, 4 human factors experts, and 2senior engineering discipline managers, all within the samedefence organisation but in different business units, ranging fromaerospace to underwater:

    A user-dened constraint: If the tool needs more that a two-page manual to explain it, it will not be used. This was a unan-imous view among a target group of engineers who wereinterviewed at the beginning of the project and reects thepracticalities of engineering life; rstly, there are never enoughengineers in society, and therefore they are usually highlyunwilling to spend signicant time on training for an unknowntool that might not produce time, cost or quality benets.A business process constraint: Systems designers are alreadyfamiliar with, and may be using as a standard procedure, tech-niques such as HEART (Human Error Assessment & ReductionTechnique (Williams, 1986)) and CREAM (Cognitive Reliabilityand Error Analysis Method - (Hollnagel, 1998)) for assessing thereliability of individuals. The tool should incorporate thesetechniques, or enable the incorporation of any other equivalentcompany technique, in order to enhance the ease of acceptanceinto design processes, and to help to address the negativeaspects in the bullet-point above.A design constraint: At the conceptual stages of design, little willbe known about the individuals in the team that will beexpected to execute any new processes, apart from genericattributes. Equally, the process will be undened e perhaps just

    a single ow diagram sketched on a sheet of paper. Hence, thetool must make a minimum demand for input data by reducingto a minimum the variables involved.

    In summary, the rst constraint has been met, in the form ofa two-page user manual, though a longer version of the manual isalso available. The second constraint has been met by creatinga three-stage tool: Stage 1 collects data about the attributes of theindividuals in the team and the intercommunications deemednecessary for execution of the process; Stage 2 collects an analysisof the process organisational environment, using either HEART orCREAM, or the organisations own in-house technique for thispurpose; and Stage 3 convolves the outputs of Stages 1 and 2, anddelivers a likelihood of success, depending on the binding of theteam to the process. Fig. 2 illustrates the three stages for using thetool.

    The third constraint has beenmet by reducing to aminimum thevariables involved, and hence the input data required


View more >